M05 · optional contextual support
Compute probabilities without overflow
Use exponential ratios, logarithms, cross entropy and stable arithmetic.
Start with arithmetic, percentages and basic algebra. Open any section directly. Practice and completion here do not change any AI course requirement or the original five-lesson mathematics pilot.
1. Try a concrete case
Logits 1,0,−1 yield probabilities proportional to e,1,1/e. Adding 1,000 to every logit changes none of the exact ratios, although direct exponentiation overflows binary64.
2. Predict before changing a value
Change the common offset from 0 to 1,000. Compare direct exponentiation with the stable result. Change the target class without changing logits. Name the quantity you expect to change and one you expect to stay fixed before using the controls.
3. Experiment
Change one quantity, follow every calculation
- Stable log loss
- 0.40761
- Probability sum
- 1.00000
- Direct exp(first logit)
- 2.71828
Vertical axis: Probability. Fixed scales; 0 points clipped for display. Lines join sampled values to aid reading; they do not create extra observations. Exact numerical values are below.
| Class | Logit | Stable probability | Loss gradient |
|---|---|---|---|
| 0 | 1 | 0.66524 | -0.33476 |
| 1 | 0 | 0.24473 | 0.24473 |
| 2 | -1 | 0.09003 | 0.09003 |
The common offset cancels algebraically. Subtract the largest logit before exponentiation; compute loss from log-sum-exp instead of log of a rounded probability. Binary64 may still underflow tiny probabilities; large input rounding cannot be reversed.
Change the common offset from 0 to 1,000. Compare direct exponentiation with the stable result. Change the target class without changing logits.
Controls save locally as practice. No timers, remote services or GPU are required. Maximum numerical work is a bounded small table or 256 bootstrap resamples.
4. Understand the calculation
Start with a calculation you can inspect
Logits 1,0,−1 yield probabilities proportional to e,1,1/e. Adding 1,000 to every logit changes none of the exact ratios, although direct exponentiation overflows binary64.
Use exponential ratios, logarithms, cross entropy and stable arithmetic. The experiment changes one named quantity at a time and shows the intermediate values, rather than asking you to accept a final score. Before changing a control, write down which value you expect to move and why.
Read the formula in words and symbols
pi = exp(zi−m)/Σj exp(zj−m); loss = m−ztarget+log Σj exp(zj−m)
zi is class i’s logit, m is the largest logit, and exp is e raised to a value. log is the natural logarithm. Probabilities sum to one; loss uses the true target index.
A subscript names a coordinate, case or time index; it is not multiplication. A summation sign means add the terms for the stated indices. Parentheses identify which calculation happens first. Carry units through arithmetic: adding incompatible quantities or confusing a rate with a count changes the question.
Connect the visual and the numbers
Change the common offset from 0 to 1,000. Compare direct exponentiation with the stable result. Change the target class without changing logits. The plot uses declared fixed axes, so a new setting does not silently rescale the apparent change. Any clipped values are counted and remain available in the numerical table. Do not infer an exact curve or a new observation from a line drawn between points.
Use it in an AI experiment
These ideas support C03, C05, C06, C10. Use the return link when you entered from an AI lesson; its original section and experiment remain separate. A mathematical support calculation is a way to inspect an assumption, not evidence that a trained model generalizes.
Optional derivation and implementation depth
The gradient of log loss with respect to logit i is pi−1[i=target]. Subtracting the maximum preserves ratios while avoiding large exponentials. Tiny probabilities can still underflow; gap-first log loss avoids taking log(0). Extremely large offsets can round away differences before the stable routine receives them.
Recompute the worked example with a separate piece of arithmetic or a short program. State the domain and test a boundary case before using the formula elsewhere. Guided numerical practice, independent implementation and human review are different outcomes.
Read exponentials and logarithms first
The exponential exp(x)=e^x uses the constant e≈2.718. The natural logarithm reverses it: log(exp(x))=x, and exp(log(y))=y for y>0. Also log(ab)=log(a)+log(b), so independent likelihood products become sums of log likelihoods. The natural-log information unit is a nat; changing the logarithm base changes the unit, not the underlying probability.
For a true-class probability 1/2, negative natural log probability is about 0.693 nats. A lower assigned probability gives a larger loss. A probability can be dimensionless while a continuous density carries inverse measurement units; keep those cases separate.
Optional implementation and research depth
The gradient of log loss with respect to logit i is pi−1[i=target]. Subtracting the maximum preserves ratios while avoiding large exponentials. Tiny probabilities can still underflow; gap-first log loss avoids taking log(0). Extremely large offsets can round away differences before the stable routine receives them.
Recompute one row independently, then test a boundary case and state which assumption it probes. Passing the supplied practice case does not establish independent implementation proficiency.
5. Apply it to a different case
Practice not yet passed; independent mastery and human review are not claimed.
6. Continue when useful
Completion not recorded. Skipping retains settings and practice; it never awards completion.
Used in C03, C05, C06, C10. Browse every support topic and course connection.