Hyunae ParkROBOSKILL LAB · PHYSICS / CONTROL / EVALUATION
When a robot misses,
what should it do next?
From executing an instruction to checking the outcome and trying again.
A small block-pushing task makes physical error and recovery measurable. Explore the motivation, 24 recorded experiments, and the next improvements.
Disturbance · seed 0 · replans 4 · final error 1.43 cm
01 / MOTIVATION
Connect an AI decision to a physical outcome
In MED-RAG I evaluated responses; in agent projects I connected actions; in VisionEye I compared source footage with perception results. I wanted to extend that work to a system that checks whether a commanded action actually moves an object to its goal.
The first task is “push the red block to the right goal.” A tool can complete its planned path while the block still misses the goal. I varied friction, mass, and an applied force to test whether observing the outcome and replanning could reduce this gap.
02 / SYSTEM
From instruction to contact and observation
MuJoCo runs on CPU with a 0.002-second timestep. The controller sets robot actuator targets; gravity, friction, and contact move the blocks. Separate approach, lower, push, lift, and observe phases make each run inspectable.
The published experiment uses a rule-based parser and ground-truth simulator coordinates. An optional Ollama adapter passes only validated task specifications and has response-contract tests. Actual LLM inference and camera perception are future steps.
- 01InstructionConstrained Korean/English grammar
- 02Task contractpush / object / goal
- 03ExecuteThree-axis robot · position actuators
- 04Observe and reviseObject position → reapproach
- 05VerifyStable arrival · trajectory and events
03 / PROTOCOL
The same starting state, two execution strategies
For each condition, a seed shifts the initial block positions by up to ±2.5 cm, with matching states for both strategies. Open loop executes one full push from the initial observation. Closed loop pushes at most 9.5 cm, reads the current position, and approaches again.
Success requires staying within 4.5 cm of the goal with linear speed below 2.5 cm/s and angular speed below 0.5 rad/s for 0.3 seconds. The other block must move less than 4 cm; leaving the workspace or exceeding 24 seconds counts as failure.
| Strategy | Execution | Shared conditions |
|---|---|---|
| open_loop | Initial observation → one full-distance push | Same seed, goal, actuators, speed, and 24-second budget |
| closed_loop | Push up to 9.5 cm → observe and reapproach | Same success criteria and distractor movement limit |
- This comparison changes push segments and reapproaches as well as observation frequency. Isolating the effect of observation is a planned follow-up.
04 / MEASURED · 2026.09.15
More goals reached, with more time spent
I executed four conditions, three seeds, and two strategies. The single-push strategy succeeded in 5 of 12 runs; observation and reapproach succeeded in all 12. Mean final distance fell from 13.28 cm to 1.08 cm, while mean simulation time rose from 7.05 to 17.71 seconds.
Both strategies succeeded in all three low-friction runs. Larger differences appeared when mass and friction increased together and when a force was applied. These conditions and seeds form a small development set also used while building the controller.
OPEN LOOP
Plan once, push once
5 / 12 successful
- Mean simulation time
- 7.05 s
- Mean final goal distance
- 13.28 cm
CLOSED LOOP
Observe again, revise the path
12 / 12 successful
- Mean simulation time
- 17.71 s
- Mean final goal distance
- 1.08 cm
Sep 15, 2026 · 4 conditions × 3 seeds × 2 strategies. These 24 development runs are not a held-out evaluation. Time is the simulated task duration.
| Condition | Settings | Single push | Observe and reapproach |
|---|---|---|---|
| Nominal | Mass 80 g · friction 0.50 | 2/3 | 3/3 |
| Low friction | Mass 80 g · friction 0.12 | 3/3 | 3/3 |
| Higher mass and friction | Mass 180 g · friction 0.70 | 0/3 | 3/3 |
| Applied force | Nominal + 1.2 N along x for 0.10 s | 0/3 | 3/3 |
- These values correspond to the saved JSON and summary. Simulation time is elapsed time in the simulated world; CPU execution time is recorded separately.
05 / CASE ANALYSIS
What changed after the applied force
For seed 0 in the force-disturbance condition, both strategies received the same force at 2.868 seconds. The single push ended 28.95 cm from the goal. The observation strategy replanned four times starting at 3.874 seconds and reached a stable position 1.43 cm from the goal at 16.81 seconds.
This successful run still touched the other block. Contact was recorded on 20 physics steps, and that block moved approximately 2.97 mm. It met the current tolerance, but remains an unwanted effect for the next controller to reduce.
| Force disturbance · seed 0 | Single push | Observe and reapproach |
|---|---|---|
| Final goal distance | 28.95 cm | 1.43 cm |
| Replans | 0 | 4 |
| Simulation time | 7.05 s | 16.81 s |
| Steps with distractor contact | 0 | 20 / 0.04 s |
- The records show a missed goal and recovery with replanning. This comparison does not isolate the contributions of friction, contact, or push segmentation.
06 / VALIDATION CHANGES
Make the result worth inspecting
I checked the experiment as well as the controller: whether initial positions survived setup, whether briefly passing the goal could count as success, and whether replay artifacts were distinguishable from execution evidence.
| Issue checked | Implemented change |
|---|---|
| Recomputing model constants can reset initial positions | Save and restore qpos around mj_setConst; test repeatability and variation across seeds |
| Success based on one position and velocity sample | Require 0.3 seconds within the goal and speed limits; reject timeouts |
| Confusing simulation evidence with display replay | Record code hashes, environment, trajectories, and events in JSON; the web viewer reads recorded results |
| Success can hide contact with the other block | Record contact steps, distractor displacement, and peak force at an individual contact point |
07 / NEXT EXPERIMENTS · PROPOSED
Fix the evaluation, then expand perception and AI
The next priorities are a more controlled comparison, camera perception, and actual LLM execution. The table below proposes work not yet run, with acceptance targets to fix before those experiments.
| Priority | Experiment | Proposed acceptance target |
|---|---|---|
| 01 | A matched baseline with the same push segments, reapproaches, and time budget; 100 unseen seeds | Report success, time, and distractor contact together; isolate the added effect of updated observations |
| 02 | Estimate block positions from rendered RGB; add occlusion and lighting conditions | On held-out data: position-error p95 ≤2 cm and success drop ≤10 percentage points versus oracle coordinates |
| 03 | Run a local LLM on valid, ambiguous, and unsupported instructions | On 100 fixed instructions: task-specification accuracy ≥95% and zero executions on the refusal set |
| 04 | Train an imitation policy from expert trajectories and compare on independent conditions | Match baseline success within the same 24-second budget while reducing execution time or distractor contact |
DESIGN DECISIONS
Three principles behind the implementation
Fix the task contract
Validate the action, object, and named goal as a task specification; the controller owns coordinates and motor commands. This keeps instruction errors distinct from physical execution failures.
Evaluate arrival and stability
A frame near the goal is insufficient, so speed and dwell time are part of success. The same check includes distractor movement and timeouts.
Keep the result inspectable
Pair success counts with time, contact, failure events, and trajectories. Recorded runs can be inspected in the browser and downloaded as JSON.
SCOPE / EVIDENCE
What is measured, and what comes next
- The 24 runs belong to a small development set. The 12/12 result applies to these conditions; success in new environments is unmeasured.
- Published measurements use a rule parser, a hand-coded controller, and simulator coordinates. LLM, VLM, VLA, and reinforcement-learning model performance has not been measured.
- This is a tabletop pushing task with a three-axis Cartesian robot. Six-axis arms, grasping, and transfer to physical hardware remain future work.
- This personal project was implemented and run with automation assistance. The public website replays saved trajectories, alongside executable code and original records.