Lesson 20 of 21 in this unit
Evaluate a robotic system.
Freeze policies and account for complete episodes, failures and exposure.
By the end: Freeze policies and account for complete episodes, failures and exposure.
Prerequisites and path options
Why learn this? Evaluate complete tasks, failures and operating limits rather than reporting action prediction accuracy alone.
Start with something concrete.
A policy reaches close to the target once, then hits a wall. A final screenshot and an average position error can hide that failed task.
Try the interactive modelWhat belongs in the full-task success denominator?
Change one thing. Watch what follows.
Loading this browser experiment…
Understand what changed.
A shorter explanation, with the original worked examples available below. Both explain the same experiment and its limits.
The question
A policy reaches close to the target once, then hits a wall. A final screenshot and an average position error can hide that failed task.
Try it and inspect what changed
Save a frozen project candidate and compare all four scenario outcomes with the teacher.
Compare the learned candidate and teacher on every case. Record which disturbances recover, which collide and which simply time out. Do not discard difficult runs. With four scenarios, avoid a precise reliability claim: additional independent tasks and repeated stochastic trials answer different questions.
The calculation in words
An episode lasts thirty 0.1-second intervals unless collision terminates it. Success requires the last three positions within 0.12 m of the goal and no collision. Wall checks include the object radius. Final error, observation count, learned-policy calls and status remain separate metrics.
Keep the conclusion within the evidence
The project freezes the entire checkpoint and settings before revealing four separate parameter combinations. Training never consumes those cases. Known exposure survives import and reset: repeated inspection is practice. Because code and cases are public client-side material, this is a self-administered teaching protocol, not a secure examination.
Simulation time is not measured CPU latency. Our ideal permanent contact gives no contact-force events to count. Report that measurement as unsupported, rather than entering a fictional zero-contact result. Hardware qualification and independent human review remain separate.
You can explore every lesson directly. Completing this page, submitting a project report and demonstrating understanding on a separate case are different records. Optional code is available in the robot-learning studio; it does not increase guided lesson completion requirements.
Open the full worked examples, calculations and limitations
This is the same reference used in Technical mode. Expand it whenever you need the exact numbers, assumptions or further explanation.
Start with the task
A policy reaches close to the target once, then hits a wall. A final screenshot and an average position error can hide that failed task.
Mechanism and units
An episode lasts thirty 0.1-second intervals unless collision terminates it. Success requires the last three positions within 0.12 m of the goal and no collision. Wall checks include the object radius. Final error, observation count, learned-policy calls and status remain separate metrics.
The project freezes the entire checkpoint and settings before revealing four separate parameter combinations. Training never consumes those cases. Known exposure survives import and reset: repeated inspection is practice. Because code and cases are public client-side material, this is a self-administered teaching protocol, not a secure examination.
Work through the evidence
Compare the learned candidate and teacher on every case. Record which disturbances recover, which collide and which simply time out. Do not discard difficult runs. With four scenarios, avoid a precise reliability claim: additional independent tasks and repeated stochastic trials answer different questions.
Save a frozen project candidate and compare all four scenario outcomes with the teacher.
What this experiment can establish
Simulation time is not measured CPU latency. Our ideal permanent contact gives no contact-force events to count. Report that measurement as unsupported, rather than entering a fictional zero-contact result. Hardware qualification and independent human review remain separate.
Keep collection, fitting, frozen execution and interpretation as separate records. Save the configuration and complete result, including failures, in the robot-learning project. Report submission checks required evidence and text presence; reasoning and implementation quality need independent review.
Eight of twelve frozen episodes succeed; two collide and two time out. What success fraction should be reported?
Take this idea with you.
Simulation time is not measured CPU latency. Our ideal permanent contact gives no contact-force events to count. Report that measurement as unsupported, rather than entering a fictional zero-contact result. Hardware qualification and independent human review remain separate.
Explore related topics and the topic map
How this topic connects
Explore this topic on the interactive mapUse these connections to review a foundation or choose where to go next. You can open any topic without completing the others.
What it builds on
- Geometry and control
Prevent coordinate-frame mistakes when combining robot positions, sensor readings and targets.
Freeze policies and account for complete episodes, failures and exposure.
What it enables
- Robot-learning project Apply in a project
Combine demonstration provenance, policy comparisons and complete-task evidence in the robot-learning project; keep embodiment limits and review status explicit.