M04 · optional contextual support
Average future rewards with explicit assumptions
Compute expectation, discounted sums and support-aware importance ratios.
Start with arithmetic, percentages and basic algebra. Open any section directly. Practice and completion here do not change any AI course requirement or the original five-lesson mathematics pilot.
1. Try a concrete case
A first outcome has probability 0.25, reward 1 and next value 3. The other has probability 0.75, reward −1 and next value 0. With discount 0.5, expected reward plus continuation is 0.25×2.5+0.75×(−1)=−0.125.
2. Predict before changing a value
Change outcome probability without changing rewards, then vary discount. Inspect each branch before averaging. Extend the horizon and explain why a discount below one downweights distant rewards. Name the quantity you expect to change and one you expect to stay fixed before using the controls.
3. Experiment
Change one quantity, follow every calculation
- Expected one-step return
- -0.12500
- Sum of n unit rewards with discount
- 1.99805
- Importance weight for first outcome
- 1.00000
Vertical axis: Discount weight γ^t. Fixed scales; 0 points clipped for display. Lines join sampled values to aid reading; they do not create extra observations. Exact numerical values are below.
| Outcome | Probability | Immediate reward | Next value | Reward + discount × value |
|---|---|---|---|---|
| 0 | 0.25 | 1 | 3 | 2.5 |
| 1 | 0.75 | -1 | 0 | -1 |
Expected value averages both branches, never only the favorable one. Reward units and continuation-value units match; gamma is dimensionless. Importance ratio target/behavior requires positive behavior probability wherever target has mass. Zero support makes this correction unavailable, not zero cost.
Change outcome probability without changing rewards, then vary discount. Inspect each branch before averaging. Extend the horizon and explain why a discount below one downweights distant rewards.
Controls save locally as practice. No timers, remote services or GPU are required. Maximum numerical work is a bounded small table or 256 bootstrap resamples.
4. Understand the calculation
Start with a calculation you can inspect
A first outcome has probability 0.25, reward 1 and next value 3. The other has probability 0.75, reward −1 and next value 0. With discount 0.5, expected reward plus continuation is 0.25×2.5+0.75×(−1)=−0.125.
Compute expectation, discounted sums and support-aware importance ratios. The experiment changes one named quantity at a time and shows the intermediate values, rather than asking you to accept a final score. Before changing a control, write down which value you expect to move and why.
Read the formula in words and symbols
E[r+γV] = Σ probability×(reward+γ next value); return = Σt γ^t rt
γ is a dimensionless discount between zero and one. t indexes future steps starting at zero. The sum includes every possible branch with its probability. A value is an expectation, not a guaranteed outcome.
A subscript names a coordinate, case or time index; it is not multiplication. A summation sign means add the terms for the stated indices. Parentheses identify which calculation happens first. Carry units through arithmetic: adding incompatible quantities or confusing a rate with a count changes the question.
Connect the visual and the numbers
Change outcome probability without changing rewards, then vary discount. Inspect each branch before averaging. Extend the horizon and explain why a discount below one downweights distant rewards. The plot uses declared fixed axes, so a new setting does not silently rescale the apparent change. Any clipped values are counted and remain available in the numerical table. Do not infer an exact curve or a new observation from a line drawn between points.
Use it in an AI experiment
These ideas support C07, C10, C12, C14. Use the return link when you entered from an AI lesson; its original section and experiment remain separate. A mathematical support calculation is a way to inspect an assumption, not evidence that a trained model generalizes.
Optional derivation and implementation depth
A Bellman equation relates a value to immediate reward and continuation under a specified policy. Off-policy weighting uses target probability divided by behavior probability; if behavior has zero support where target assigns mass, the estimate is not identified by that log. Large ratios increase variation. A baseline can reduce policy-gradient variance without changing its expectation only under the stated action-independence conditions.
Recompute the worked example with a separate piece of arithmetic or a short program. State the domain and test a boundary case before using the formula elsewhere. Guided numerical practice, independent implementation and human review are different outcomes.
Optional implementation and research depth
A Bellman equation relates a value to immediate reward and continuation under a specified policy. Off-policy weighting uses target probability divided by behavior probability; if behavior has zero support where target assigns mass, the estimate is not identified by that log. Large ratios increase variation. A baseline can reduce policy-gradient variance without changing its expectation only under the stated action-independence conditions.
Recompute one row independently, then test a boundary case and state which assumption it probes. Passing the supplied practice case does not establish independent implementation proficiency.
5. Apply it to a different case
Practice not yet passed; independent mastery and human review are not claimed.
6. Continue when useful
Completion not recorded. Skipping retains settings and practice; it never awards completion.
Used in C07, C10, C12, C14. Browse every support topic and course connection.