World Action Agent: VLMs Rehearse Robot Moves Before Physical Execution

A new robot manipulation framework called the World Action Agent, published this week as a preprint on arXiv, proposes a different answer to one of embodied AI's persistent challenges: how do you get a general-purpose vision-language model to reliably control a robot arm when it was never trained to do so? Rather than building a specialized end-to-end model from robot demonstrations or generating symbolic code for the robot to execute, the World Action Agent's approach is to show the robot what it is about to do — and let it revise and confirm — before any physical movement occurs. Evaluated on LIBERO-Pro, a harder robustness variant of the LIBERO tabletop manipulation benchmark, the framework reaches 75.6% task success. When the framework's interaction logs are then used to fine-tune a 9-billion-parameter open-weight model, that model's out-of-domain success rate rises from 1.7% to 43.3%, pointing toward local deployment without reliance on large cloud APIs at inference time.

The preprint was submitted to arXiv on September 24, 2026, and surfaced on HuggingFace Daily Papers the following day. The paper carries a "working in progress" label from its 16-person authorship team, whose affiliations include HKUST(GZ) and CUHK: results have not been independently reproduced, and the framework has been evaluated in simulation rather than on physical hardware. The preprint's early release is consistent with the current norm in AI research of rapid community circulation, but it also means the results arrive without the additional baselines and ablation studies that peer review typically requires.

A Visual Workspace That Structures What a Robot Can See and Edit

The framework's core construct is what the authors call the Visual Action Workspace — three design choices that together change how a VLM interacts with a manipulation task.

The first element is contact views: viewpoints automatically selected from the scene's 3D geometry to place the VLM at angles that minimize occlusion and provide clear sight lines to gripper-object contact regions. This addresses a frequently underappreciated failure mode in VLM-based robotics, where the model misidentifies object positions because the camera is in a poor position relative to the action.

The second element is action rehearsal, the framework's most distinctive contribution. Before any motor command is issued, an internal sub-agent called the Imagination Agent generates proposed action trajectories rendered as a visual overlay in the workspace — a 3D preview of what the robot intends to do. The VLM reviews this representation and can edit or reject the proposal before execution. The mechanism resembles a person sketching a path with their finger before reaching for an object: the visualization creates a self-checking loop that lets the model catch spatial or sequencing errors it would miss from text instructions alone. The authors call this capacity World Action Rehearsal, positioning it within an emerging category of systems that explicitly model future world states before committing to action.

The third element is in-view correction: after an action executes, corrections are applied within the same visual workspace rather than through a separate feedback channel, keeping the perceptual loop coherent across the observe-plan-act-correct cycle.

Skill Evolution from Demonstrations — Not Zero-Shot Learning

An earlier news brief described the WAA as achieving "zero-shot" robot control — a framing the actual architecture does not support. The framework's second sub-agent, the Skill Agent, consults a library of multimodal skills evolved from expert video demonstrations drawn from the LIBERO-90 dataset: a set of 90 tabletop manipulation tasks with human demonstrations. These skills are refined through an "evidence-based review" process rather than replayed directly, but their origin in demonstrations means WAA is not zero-shot in the classical sense of requiring no examples.

What the framework achieves is cross-task generalization: skills evolved on LIBERO-90 transfer to LIBERO-Pro and LIBERO-Plus — a harder robustness benchmark and 10,030 task variations respectively — without retraining for those targets. Standard end-to-end VLA training often degrades significantly when the task distribution changes; WAA, by routing through a general-purpose VLM with a structured workspace, appears to reduce that sensitivity. The distinction matters practically: teams evaluating robot learning systems need to know whether they are comparing against a zero-shot baseline or a system that requires demonstrations at the task-class level — which is WAA's actual requirement.

Read more: Frontier VLMs Control Robots Zero-Shot Through a Compact Semantic Action Interface

What the LIBERO-Pro Result Shows — and What It Does Not

LIBERO-Pro tests robustness across distractor objects, background changes, and modified goal specifications — conditions more representative of real task variation than the cleaner LIBERO base suite. WAA's 75.6% is reported as favorable relative to two baseline categories: Code-as-Policy systems that have a VLM generate Python programs for pre-defined robot primitives, and end-to-end VLA models trained on robot demonstration data.

The comparison with Code-as-Policy approaches is informative because it partially isolates what the visual rehearsal step contributes. Code-as-Policy systems also use a general-purpose LLM or VLM as their reasoning layer, but they decompose manipulation into discrete symbolic programs executed against fixed motion primitives — for example, a function call like grasp(object="cup") with a hard-coded gripper motion. WAA's visual workspace operates over continuous motion trajectories rendered in 3D, which may give the VLM more geometrically meaningful information during plan verification: rather than approving an abstract function name, the model reviews a rendered trajectory and can catch spatial errors before execution. The claim is plausible on its face but would benefit from an ablation that removes the visual rehearsal step to isolate its specific contribution to the benchmark result.

Comparison against π0/π0.5 from Physical Intelligence and NVIDIA's GR00T N1 represents a different dimension: those end-to-end VLAs are trained on large corpora of robot demonstrations and optimized for specific embodiments. They are not being directly supplanted by WAA's simulation results; the relevant question for deployment teams is whether cross-task generalization in simulation predicts real-world cross-task generalization — which remains open.

The Distillation Finding: From Harness to Deployable Small Model

The result with the clearest practical implication is the trace-based distillation experiment. The WAA harness generates logs of its own operation — the visual observations, action rehearsal steps, corrections, and skill consultations that compose each interaction — and the research team used these logs to fine-tune Qwen3.5-9B, an open-weight 9-billion-parameter VLM from Alibaba's Qwen group. The choice of Qwen3.5-9B is significant: it is a relatively small, open-weight model that can run locally on consumer-grade GPU hardware, rather than a proprietary frontier model. Out-of-domain task success rose from 1.7% to 43.3%.

The 1.7% baseline illustrates how poorly an unmodified general-purpose VLM performs on manipulation tasks without adaptation. The endpoint — 43.3% — is below production-capable thresholds, but the more-than-25-fold improvement demonstrates that the harness's interaction traces carry substantial training signal. The practical architecture this suggests is two-stage: run the full WAA harness in simulation to generate high-quality interaction data, then distill that data into a smaller deployable model that runs locally without cloud inference at execution time. Physical robots operating in warehouses or in connectivity-constrained environments face real per-inference API latency and cost constraints that make this distillation path relevant. Whether 43.3% generalizes to physical hardware is a separate question the paper does not address, but the magnitude of improvement from traces alone establishes the approach as worth pursuing.

VLM Harnesses vs. End-to-End VLAs: Two Competing Philosophies

The World Action Agent enters a field where two strategies are contending for momentum. End-to-end VLA models — π0, GR00T N1 — internalize manipulation physics through robot-specific training data and deliver strong performance on the tasks they are trained for. VLM harness approaches — WAA, Show-Harness, Code-as-Policy — argue that general-purpose VLMs already carry sufficient reasoning and visual understanding to be leveraged through structured interfaces, reducing dependence on large robot-specific datasets.

Within the harness camp, WAA's position differs from Show-Harness, which was published by the NUS Show Lab in a separate preprint earlier in September 2026 and covered separately by this publication. Show-Harness decomposes manipulation into 12 discrete semantic action units — reach, grasp, place, and similar categories — and routes VLM reasoning through that fixed vocabulary, reporting 86–96% zero-shot success on its evaluation tasks. The two systems are not directly comparable: they run on different benchmarks with different task distributions, and they embody different structural bets. Show-Harness decomposes by action semantics, making the vocabulary of manipulation explicit; WAA decomposes by visual rendering and review, keeping the action trajectory continuous but making it spatially visible before execution. Both approaches add structure that the underlying VLM does not possess natively; they differ in what dimension of structure they impose. The existence of both in the same month of September 2026 suggests the harness design space is being actively explored from multiple directions simultaneously.

WAA's rehearsal loop also connects to a broader trend in AI toward structured self-verification, where model outputs improve when the model evaluates a generated plan before committing. Chain-of-thought prompting in language reasoning, test-driven code generation, and iterative mathematical proof verification all reflect the same principle. WAA applies it to spatial action planning, where the verification is rendered in a visual format matched to the task domain.

Read more: Technion Researchers Add Probability to Robot Vision Grounding to Fix a Core Planning Flaw

Limitations That Matter Before Treating These Results as Deployment Benchmarks

The simulation-to-real gap is the most significant qualification on WAA's results. LIBERO-Pro is entirely simulation-based, running in a physics engine that does not fully capture the contact dynamics, sensor noise, friction variation, or object deformation that arise when a real robot arm interacts with physical objects. Systems that perform well in LIBERO-style environments sometimes fail substantially in hardware trials because simulation imposes implicit simplifications — perfect gripper compliance, idealized object geometry, predictable mass distribution — that do not hold in practice. Until the framework is evaluated on physical hardware and reproduced by an independent research group, the 75.6% figure should be read as a promising simulation result rather than a deployment benchmark.

The distilled Qwen3.5-9B model at 43.3% is far below what a production deployment would require, and the quality and quantity of traces needed to reach deployment-grade performance is unknown. The skill library's dependence on LIBERO-90 demonstrations also ties the current implementation to those tasks' object categories and manipulation styles; how well the skill evolution mechanism would generalize to substantially different domains — surgical robotics, construction assembly, warehouse logistics with non-standard objects — remains untested.

The "working in progress" label on the preprint is a genuine flag: robotics AI papers can shift substantially between early release and peer review as additional baselines, ablation studies, or real-world experiments reveal gaps in the original evaluation scope.

What the Preprint's Early Release Suggests About Next Steps

The decision to release early — with a "working in progress" label — suggests the team sees the current framework as a base to extend rather than a finished system. Physical hardware evaluation is the most important missing validation. Broadening the skill library beyond LIBERO-90 to datasets like Open X-Embodiment would test whether skill evolution is specific to simulation or more general. Scaling the distillation experiment with more traces and larger target models would establish where the distillation curve flattens.

The deeper question the framework opens is whether action rehearsal becomes more or less important as underlying VLM spatial reasoning improves. If future models carry richer internal representations of 3D physical dynamics, the explicit visual workspace may matter less; if spatial reasoning remains a relative weakness even at larger scale, the external scaffold may remain load-bearing. Either answer is useful, and the way to find it is to run the rehearsal loop against progressively stronger backbone models — a natural experimental track for the group and for others who will reproduce or extend the work.