Predict in a compact representation.

This is the same reference used in Technical mode. Expand it whenever you need the exact numbers, assumptions or further explanation.

Pictures, codes and action predictions

A latent representation is a learned internal description rather than the original measurement. Here an encoder compresses a sixteen-pixel picture into two numbers. A predictor receives those two numbers and an action, then predicts the next picture’s code. A decoder can map a code back to pixels, but predicting a useful code does not inherently require generating a picture.

The generated scene is a soft spot moving horizontally in a 4×4 frame. Actions −1, 0 and +1 move its horizontal position by −0.18, 0 and +0.18 scene units; the position is clipped to ±0.9. Optional brightness and checker texture stay fixed within each episode. They change appearance without changing the motion rule.

Eight six-step training episodes supply 48 pairs. Four separate development episodes supply 24. Whole episodes are assigned before training, and adjacent frames remain correlated. These are tiny authored scenes, not camera data or evidence about arbitrary videos.

I denotes an image, z its two-number code and u the action. The same encoder processes both frames. Gradients from both sides accumulate into its shared weights. The predictor sees neither the future image nor the simulator’s position labels.

Why a low error can mean nothing

Suppose every image becomes [0, 0]. A predictor that always returns [0, 0] perfectly matches every next code, even when the physical object moves. This is representation collapse: different observations lose their distinctions.

Prediction-only training minimizes the mean squared difference between predicted and next-image codes. Both encoder outputs are allowed to change. The loss can decrease by making motion easier to predict, but it can also decrease by making all targets similar. The collapsed control in the lab actually sets all weights to zero; its zero loss is computed, not a fabricated score.

Compare three concrete objectives

Let L be mean squared next-code error across pairs and both code dimensions. Let R be mean squared reconstruction error across both images in each pair and all sixteen pixels. Reconstruction-plus-prediction trains the encoder, predictor and sigmoid pixel decoder to minimize L + 10R. The factor ten is an authored scale choice for these small normalized images, not a universal recipe.

Prediction-only minimizes L and leaves the decoder inactive. The third mode adds explicit terms that discourage constant codes and redundant coordinates. It also leaves the pixel decoder inactive.

To calculate V, pool the 96 source/target training encodings. For each coordinate, find its average, average the squared distances from that average, add 0.0001 and take the square root. This is the displayed regularized spread. Penalize the squared amount by which it falls below 0.3, then average over the two coordinates. A perfectly constant code therefore reports 0.01 rather than zero because of the added number.

C is the square of the off-diagonal population covariance: average the product of the two centered coordinates, then square it. It discourages the two coordinates from varying together. Low covariance alone does not establish independence or useful task information.

Here M = 96 during training. All moments use divisor M. This is an action-conditioned teaching experiment inspired by variance/covariance regularization, not a replication of the VICReg paper. It uses squared shortfall, pooled temporal views and a learned action predictor with its own specified weights and limits.

A penalty is not a guarantee

The exact zero control has L = 0, V = 0.29² and C = 0. The spread objective is therefore positive. Yet every centered code is zero, so the corresponding gradients at that perfectly symmetric point are zero too. Reset to a nonconstant seeded initialization to explore learning. The objective discourages collapse without guaranteeing escape from every initialization.

Even varied codes can encode the wrong thing. Static texture is easy to predict because it stays unchanged. With two coordinates and several sources of variation, the model can trade motion information for appearance information. Healthy spread, low prediction error and visually plausible reconstruction are not interchangeable evidence.

Inspect information without leaking it into training

A separate diagnostic readout asks what a simple affine rule can recover from the frozen codes. It fits position and pixels on the 96 training encodings only, with ridge 0.0001 on its two slopes and no penalty on its intercept. Position labels never update the encoder or predictor. Re-fitting this readout after a checkpoint is diagnostic work, not an additional representation-training step.

The same readout evaluates development encodings and predicted next codes. Compare current pixel error with a training-mean image; current-position error with next-position error; and action-conditioned next-position error with keeping the current code. A limited affine readout can miss information that a more flexible decoder could recover, so its error is not proof that all information is absent.

The neural pixel decoder’s error is reported separately and only in reconstruction mode. Its absence in the other modes must not be mistaken for a failed trained decoder. Image previews from the diagnostic readout clip brightness to 0–1, count clipped values, and preserve exact unclipped values in metrics and exports. Codes use fixed ±1 plot axes; latent coordinates are not physical coordinates.

Make a controlled comparison

Use the same initialization and update budget across objectives. Add distractors as a separate intervention. Change the probe action while holding the observed frame fixed, and inspect the true next frame, predicted code and position readout. These counterfactual probes never train the representation.

The encoder has 34 parameters, the action predictor eight and the decoder 48. Simultaneous full-batch SGD uses rate 0.3, at most 25 updates per worker batch and 1,000 total. Larger requests commit complete batches and can be cancelled. One direct CPU update remains available. Six temporary captures retain settings, weights, diagnostics and bounded notes; reload clears them unless exported.

This lesson studies one-step representations. It does not yet estimate hidden state across occlusion, train a policy in imagined trajectories, or establish independent proficiency. The fixed development examples and diagnostic labels are public practice; later project assessment and human implementation review remain distinct.