Lesson 17 of 21 in this unit
Connect vision language and action.
Trace finite instruction features through a learned action policy.
By the end: Trace finite instruction features through a learned action policy.
Prerequisites and path options
Why learn this? Distinguish finite grounded instruction inputs from unrestricted language understanding in an action policy.
Start with something concrete.
The same puck position needs a different action when the requested destination changes. A policy needs both scene information and the task specification.
Try the interactive modelDoes a two-token instruction policy understand arbitrary language?
Change one thing. Watch what follows.
Loading this browser experiment…
Understand what changed.
A shorter explanation, with the original worked examples available below. Both explain the same experiment and its limits.
The question
The same puck position needs a different action when the requested destination changes. A policy needs both scene information and the task specification.
Try it and inspect what changed
Change left to right, inspect goal coordinates and compare teacher and learned outputs.
Draw three boxes and annotate which are programmed and which are learned in this lab. Then compare the linked OpenVLA architecture: visual encoders, a projector, a language-model backbone and decoded action tokens. Identify a dataset or hardware claim that our tiny experiment cannot test.
The calculation in words
The instruction is represented by two indicators: right is [1,0] and left is [0,1]. Puck and goal coordinates occupy the other four inputs. Tanh hidden features combine these values and output bounded velocities. A programmed dispatcher chooses the goal side; the motor mapping is learned from demonstrations.
Keep the conclusion within the evidence
Separate three links: scene to representation, instruction to task, and task-conditioned proposal to execution. Here the renderer supplies exact positions and the dispatcher handles only two tokens. Neither performs open-ended perception or language understanding. Unsupported commands are outside the contract rather than silently guessed.
OpenVLA (2024 project, reviewed 23 October 2026) reports a pretrained 7B vision-language-action model and robot evaluations. Published model behavior and this finite-vocabulary simulation are different evidence; do not transfer the paper’s results to the browser policy.
You can explore every lesson directly. Completing this page, submitting a project report and demonstrating understanding on a separate case are different records. Optional code is available in the robot-learning studio; it does not increase guided lesson completion requirements.
Open the full worked examples, calculations and limitations
This is the same reference used in Technical mode. Expand it whenever you need the exact numbers, assumptions or further explanation.
Start with the task
The same puck position needs a different action when the requested destination changes. A policy needs both scene information and the task specification.
Mechanism and units
The instruction is represented by two indicators: right is [1,0] and left is [0,1]. Puck and goal coordinates occupy the other four inputs. Tanh hidden features combine these values and output bounded velocities. A programmed dispatcher chooses the goal side; the motor mapping is learned from demonstrations.
Separate three links: scene to representation, instruction to task, and task-conditioned proposal to execution. Here the renderer supplies exact positions and the dispatcher handles only two tokens. Neither performs open-ended perception or language understanding. Unsupported commands are outside the contract rather than silently guessed.
Work through the evidence
Draw three boxes and annotate which are programmed and which are learned in this lab. Then compare the linked OpenVLA architecture: visual encoders, a projector, a language-model backbone and decoded action tokens. Identify a dataset or hardware claim that our tiny experiment cannot test.
Change left to right, inspect goal coordinates and compare teacher and learned outputs.
What this experiment can establish
OpenVLA (2024 project, reviewed 23 October 2026) reports a pretrained 7B vision-language-action model and robot evaluations. Published model behavior and this finite-vocabulary simulation are different evidence; do not transfer the paper’s results to the browser policy.
Keep collection, fitting, frozen execution and interpretation as separate records. Save the configuration and complete result, including failures, in the robot-learning project. Report submission checks required evidence and text presence; reasoning and implementation quality need independent review.
Our renderer supplies coordinates and the dispatcher accepts left/right. Which component is actually learned?
Take this idea with you.
OpenVLA (2024 project, reviewed 23 October 2026) reports a pretrained 7B vision-language-action model and robot evaluations. Published model behavior and this finite-vocabulary simulation are different evidence; do not transfer the paper’s results to the browser policy.
Explore related topics and the topic map
How this topic connects
Explore this topic on the interactive mapUse these connections to review a foundation or choose where to go next. You can open any topic without completing the others.
What it builds on
- Geometry and control
Prevent coordinate-frame mistakes when combining robot positions, sensor readings and targets.
Trace finite instruction features through a learned action policy.
What it enables
- Robot-learning project Apply in a project
Combine demonstration provenance, policy comparisons and complete-task evidence in the robot-learning project; keep embodiment limits and review status explicit.