Learn reusable temporally extended skills.

This is the same reference used in Technical mode. Expand it whenever you need the exact numbers, assumptions or further explanation.

A skill includes knowing when to stop

The grid task asks an agent to visit cell 0 and then cell 5. Rather than choosing every movement with one policy, a controller can call a “reach 0” skill followed by a “reach 5” skill. The high-level order is programmed here. The low-level action values and termination probabilities are learned from actual simulator transitions.

An option consists of an initiation condition, an internal policy and a termination rule. This teaching example permits either option from any cell, chooses a primitive action from its learned Q values, and stops when its learned termination probability reaches 0.8. A safety budget still limits execution even if the learned rule never stops.

Trace a real update

The exact environment is a 2×3 grid. Actions up, right, down and left move one cell; boundary actions remain in the current cell. The option receives reward 1 on reaching its waypoint and −0.05 otherwise. Its Q-learning target is reward plus 0.9 times the best next-state value, except that a true terminal arrival has no bootstrap term.

Suppose the old action value is zero, the next value is 0.4 and the move is nonterminal. The target is −0.05 + 0.9 × 0.4 = 0.31. With learning rate 0.3, the new value is 0.093. Those values are estimates used to choose movement; they are not observed success rates.

Termination uses one logit for each option/cell pair. Its sigmoid converts that logit to a number between zero and one. An exact arrival label supplies the supervised target. Logistic cross-entropy changes the logit by 0.4 times (arrival label minus predicted probability). From a zero logit, a true arrival changes it to 0.2, whose probability is about 0.550. Several consistent examples are needed before the 0.8 threshold is crossed.

Compare an honest set of baselines

The flat learner has a table indexed by task stage and cell. It learns the whole two-waypoint sequence, receiving 0.2 for the first waypoint, 1 for the last and −0.05 otherwise. The fixed hierarchy follows a known shortest-path rule and does not learn. It is a serious control, not a deliberately random opponent.

Both learned controllers receive at most twelve environment transitions per training episode, but their episode lengths can differ. Report both actual transition counts. Equal episode counts do not match interaction work, and different rewards matter when interpreting efficiency. Training caps at 120 episodes and a click executes at most five episodes on CPU; there is no background training timer.

Separate termination and task success

Evaluation never updates Q values or logits. Inspect the complete trace for each of four starts. A learned option might stop in the wrong place, fail to stop at the goal, or spend its entire budget repeating an action. These are different failures. The 24-decision execution ceiling is a runtime safeguard, not evidence that a termination policy was learned correctly.

This is tabular learning on one small, repeatedly exposed map. It is not a neural option policy, learned task decomposition or demonstrated transfer to a new environment. Extend and test one mechanism in the adaptation studio while retaining the exact environment and a declared comparison budget.