Improve a policy using imagined experience.

This is the same reference used in Technical mode. Expand it whenever you need the exact numbers, assumptions or further explanation.

Practice inside a prediction

Suppose each real trial costs time. You collect transitions, fit a model and use its predictions to try different actions. The attraction is reuse: one collection supports many imagined trajectories. The risk is that optimization discovers actions where the fitted model is misleading.

The lab reuses the exact velocity-first cart with 0.1-second intervals, mass 1 kg, linear drag 0.2 and optional quadratic drag 0.6. An affine delta model fits the 96 measured training transitions. It is not handed the plant coefficients. Its weights stay frozen during policy improvement.

What changes during learning?

A two-weight policy chooses force $u=2\tanh(w_1x+w_2v)$. Here x is position in metres, v is velocity in metres per second, and u is force in newtons; inputs use reference scales of 1 m and 1 m/s. The output always lies between −2 and +2 N. Initially both weights are zero, so no force is applied.

The objective averages $x^2+0.1v^2+0.01(u/2)^2$ after each predicted interval, using those reference units. The first term encourages reaching the origin; the second discourages residual speed; the third prices effort. These are declared design choices, not universal safety criteria.

For each weight, the optimizer evaluates the full trajectory cost at a small positive and negative perturbation. Their difference divided by twice the perturbation approximates the derivative. Both weights update from the same preceding policy, with rate 0.3 and bounds ±3. No imaginary state is silently replaced by a true state within a rollout.

A comparison with an explicit budget

One learner uses the fitted dynamics; another uses the exact simulator during optimization. They share four training starts, zero initialization, horizon and update count. Each update evaluates four candidate weight vectors, each on four starts: sixteen trajectories. At horizon five this is 80 transition calls per update. The simulator-trained comparator pays those calls to the environment; the imagined learner pays them to the model, plus initial data collection and fitting.

Both frozen policies then execute 20 exact intervals at two different public development starts. The initial zero policy is also evaluated. Evaluation uses 120 environment calls total. Fitting arithmetic and other overhead are additional; call counts are not measured time or claims of equal hardware cost.

Short imagination can still be wrong

Compare horizons five and fifteen at matched update counts. A longer rollout makes future consequences visible but also compounds model error. Gentle collection can leave action effects weakly represented. A lower imagined objective does not by itself support an external improvement claim.

The browser recomputes a bounded deterministic comparison, at most forty updates, on request. Exports retain recipes, policies, trajectories and counts. This two-weight experiment teaches the separation between data, model, policy and evaluator; it is not a reproduction of large latent actor–critic systems such as the named DreamerV3 paper.