M11 · optional contextual support
Notice the winner selected by chance
Inspect repeated comparisons, weak signals and predeclared decision rules.
Start with arithmetic, percentages and basic algebra. Open any section directly. Practice and completion here do not change any AI course requirement or the original five-lesson mathematics pilot.
1. Try a concrete case
Twenty independent null comparisons each have a 5% false-alarm rate. The chance of at least one false alarm is 1−0.95²⁰, about 64.15%, even though no tested effect is real.
2. Predict before changing a value
Increase comparisons while keeping their individual rate fixed. Increase measurement noise and inspect the same effect relative to its standard error. Write a decision rule before comparing candidates. Name the quantity you expect to change and one you expect to stay fixed before using the controls.
3. Experiment
Change one quantity, follow every calculation
- Chance of ≥1 false alarm under independence
- 0.64151
- Expected false alarms
- 1.00000
- Standard error with 20 independent observations
- 0.22361
- Effect 0.2 / standard error
- 0.89443
Vertical axis: At least one false alarm. Fixed scales; 0 points clipped for display. Lines join sampled values to aid reading; they do not create extra observations. Exact numerical values are below.
| Comparisons | Chance at least one false alarm |
|---|---|
| 1 | 0.05000 |
| 5 | 0.22622 |
| 10 | 0.40126 |
| 20 | 0.64151 |
| 50 | 0.92306 |
Assumes each null test has false-alarm rate α and is independent. Correlated searches need different accounting. Larger noise makes the same effect harder to resolve; this signal/SE ratio is not a p-value or power calculation. Predeclare the decision rule and report every tried comparison.
Increase comparisons while keeping their individual rate fixed. Increase measurement noise and inspect the same effect relative to its standard error. Write a decision rule before comparing candidates.
Controls save locally as practice. No timers, remote services or GPU are required. Maximum numerical work is a bounded small table or 256 bootstrap resamples.
4. Understand the calculation
Start with a calculation you can inspect
Twenty independent null comparisons each have a 5% false-alarm rate. The chance of at least one false alarm is 1−0.95²⁰, about 64.15%, even though no tested effect is real.
Inspect repeated comparisons, weak signals and predeclared decision rules. The experiment changes one named quantity at a time and shows the intermediate values, rather than asking you to accept a final score. Before changing a control, write down which value you expect to move and why.
Read the formula in words and symbols
P(at least one false alarm)=1−(1−α)^m; standard error=σ/√n
α is each test’s false-alarm probability under its null model, m is the number of independent comparisons, σ is outcome standard deviation, and n is the number of independent observations.
A subscript names a coordinate, case or time index; it is not multiplication. A summation sign means add the terms for the stated indices. Parentheses identify which calculation happens first. Carry units through arithmetic: adding incompatible quantities or confusing a rate with a count changes the question.
Connect the visual and the numbers
Increase comparisons while keeping their individual rate fixed. Increase measurement noise and inspect the same effect relative to its standard error. Write a decision rule before comparing candidates. The plot uses declared fixed axes, so a new setting does not silently rescale the apparent change. Any clipped values are counted and remain available in the numerical table. Do not infer an exact curve or a new observation from a line drawn between points.
Use it in an AI experiment
These ideas support C07, C15. Use the return link when you entered from an AI lesson; its original section and experiment remain separate. A mathematical support calculation is a way to inspect an assumption, not evidence that a trained model generalizes.
Optional derivation and implementation depth
Power is the chance a declared test rejects under a specified alternative; it requires a test distribution and effect assumption. The lab’s signal/SE ratio is not a power estimate. Adaptive model selection uses information from evaluation; preserve all choices and use an appropriate final protocol. A p-value is neither the probability the hypothesis is true nor a measure of practical importance.
Recompute the worked example with a separate piece of arithmetic or a short program. State the domain and test a boundary case before using the formula elsewhere. Guided numerical practice, independent implementation and human review are different outcomes.
Optional implementation and research depth
Power is the chance a declared test rejects under a specified alternative; it requires a test distribution and effect assumption. The lab’s signal/SE ratio is not a power estimate. Adaptive model selection uses information from evaluation; preserve all choices and use an appropriate final protocol. A p-value is neither the probability the hypothesis is true nor a measure of practical importance.
Recompute one row independently, then test a boundary case and state which assumption it probes. Passing the supplied practice case does not establish independent implementation proficiency.
5. Apply it to a different case
Practice not yet passed; independent mastery and human review are not claimed.
6. Continue when useful
Completion not recorded. Skipping retains settings and practice; it never awards completion.
Used in C07, C15. Browse every support topic and course connection.