Frontier VLMs Control Robots Zero-Shot Through a Compact Semantic Action Interface

A research team at the National University of Singapore's Show Lab published a paper this week arguing that the dominant bottleneck in deploying AI on physical robots is not model intelligence or training data volume — it is interface design. Their system, Show-Harness, uses a vocabulary of twelve discrete symbolic actions to bridge frozen frontier vision-language models to real robot hardware, enabling zero-shot manipulation without any embodiment-specific fine-tuning. In head-to-head experiments run by the authors, the approach outperformed dedicated vision-language-action models including Physical Intelligence's π0.5 and NVIDIA's GR00T N1 across ten manipulation tasks, four environment shifts, and two robot platforms. The paper, arXiv:2609.10522, was submitted September 9, 2026, with code and models released publicly the following day.

The paper arrives alongside a public code repository, six LoRA model adapters, and a demonstration dataset on HuggingFace, as the robotics industry grapples with the cost of producing the labeled physical interaction data that purpose-built robot AI systems require. Standard vision-language-action pipelines typically demand hundreds or thousands of teleoperated demonstrations — human operators physically guiding a robot arm while sensors record every joint angle, gripper force, and camera frame — before a model can generalize to a target task. Show-Harness's zero-shot mode bypasses that requirement entirely: a closed-source frontier VLM receives a camera feed and task instructions, and outputs one twelve-symbol action at a time, with a separate interpreter converting each symbol into actual robot motion.

How the Semantic Action Space Works

The core of Show-Harness is a fixed vocabulary of twelve discrete action units: MV_FWD, MV_BACK, MV_LEFT, MV_RIGHT, MV_UP, MV_DOWN, ROTATE_CW, ROTATE_CCW, GRASP, RELEASE, STILL, and DONE. These are not coordinates, trajectories, or Python functions — they are named symbols the model emits one per reasoning step. The embodiment-specific interpreter, a deterministic module the researchers write separately for each robot platform, translates each symbol into metric motion. Switching from a Franka 7-DoF arm to an AgileX Piper requires changing only the interpreter and its configuration file; the model, its prompts, and the twelve-symbol vocabulary stay unchanged.

The design decision to make action units symbolic and named — rather than arbitrary labels like A through F — is load-bearing, not cosmetic. An ablation reported in the paper found that using arbitrary symbols with the correct directional convention still reaches 95% task success, while using the semantic names without fixing the convention falls to 90%. Remove both — leaving the model to infer the mapping from its own trial-and-error — and success collapses to 5%, with 19 of 20 rollouts timing out. The interface is not decoration around the action space; it constitutes the action space. The paper's authors write that the model already encodes much of what robot manipulation requires — spatial reasoning, object recognition, goal decomposition — and that the semantic interface provides the last mile of grounding needed to express those capabilities as robot motion.

Each control step closes a perception-reasoning-action loop. A plugin system assembles the VLM's input context: camera images from one or more viewpoints, optional wrist-camera data, proprioception (joint states and end-effector pose), a subtask plan generated at the start of the episode, and a rolling action history. The VLM then emits a single symbol. The interpreter executes it. The loop repeats until DONE is emitted or a time limit is reached. The plugin components are modular and togglable: multi-view guidance accounts for 38 percentage points in ablation studies, subtask planning for 36 points, and proprioception for 28. An action-chunking plugin can batch multiple execution steps before the next VLM call, reducing API calls by 19% while holding success rates constant.

Five Frontier VLMs Tested — Spatial Grounding Is the Gap, Not Planning

The paper evaluated five frontier VLMs operating zero-shot: Gemini-3.1 Pro, GPT-5.6-sol, Opus 5, GPT-5.6-luna, and Gemini-3.6 Flash — model names as reported by the paper's authors. The team reports these models split into two performance tiers: Gemini-3.1 Pro, GPT-5.6-sol, and Opus 5 achieving 86–96% success across the benchmark suite, while GPT-5.6-luna and Gemini-3.6 Flash cluster at 72–78%.

A capability decomposition experiment on a chess-cannon manipulation task — a precision placement problem — identified where the performance gap originates. Instruction following and subtask planning were nearly saturated across all five models, reaching 97–98% accuracy. The discriminating factor was fine-grained spatial grounding: the top-tier models showed empty-grasp rates (missed or misaligned grippers) of 6–15%, while lower-tier models climbed to 26%. The implication is that frontier VLM progress relevant to robot control is currently limited not by language understanding or goal reasoning but by the models' ability to precisely localize objects in 3D space.

Increasing the thinking budget helped performance but at a cost. GPT-5.6-sol with higher thinking effort averaged 30 steps per episode versus 37 at lower effort — shorter episodes indicate more efficient action selection — but consumed 3.4 times the wall-clock time per episode due to additional reasoning compute. For real-time deployment scenarios, this tradeoff may be impractical.

When Fine-Tuning Beats Frontier Models on Latency

The zero-shot mode depends on frontier VLMs accessed via API, which introduces latency that can exceed what fast-moving tasks require. To address this, Show-Harness includes a fine-tuned mode: a small open-weight VLM, trained on GUMI-collected demonstrations using LoRA adapters via LLaMA-Factory and served locally with vLLM, emits one action token per step without the full planning loop. The default fine-tuned configuration uses Qwen3.5-2B, which reaches 33 Hz on real hardware; the 9B variant runs at approximately 12 Hz.

The scaling behavior is non-linear in ways that matter for deployment choices. Success on standard manipulation tasks is flat from 2B through 9B — Qwen3.5 at 2B, 4B, and 9B all fall within four percentage points of each other (88–92%). The smallest backbones (0.8B) technically emit valid action units but hesitate, stretching episodes from 37 to 58 steps at 72–74% success — long enough to increase collision risk and fail on time-limited tasks. Where the 9B model loses is on dynamic targets: a rolling tennis ball task shows 2B at 80%, 0.8B at 70%, and 9B at 60%. At 79 milliseconds per decision the larger model cannot track a moving object, whereas the 2B model's 39ms response keeps pace. For static manipulation tasks requiring geometric precision — block stacking, peg insertion — the 9B model's advantage in spatial grounding raises success from 50% to 75%; for anything time-sensitive, 2B is the better choice.

The team also released six LoRA adapters on HuggingFace (for Qwen3.5 at 0.8B, 2B, 4B, and 9B, plus Gemma 4 e4b and a simulation-only Qwen3.5-2B policy), along with the real-robot demonstration corpus covering Franka and AgileX Piper rollouts alongside ManiSkill simulation data.

GUMI Replaces Teleoperation Hardware With a Browser and a Keyboard

The data collection problem for fine-tuning is addressed by GUMI (GUI Manipulation Interface), which maps the twelve action units to keyboard keys in a browser interface. A human demonstrating a task presses WASD or arrow keys; each keypress is recorded as a (observation, action) training pair ready for LoRA fine-tuning. Because every action unit is a labeled symbol rather than a continuous motion command, the same browser interface can drive a GUI agent — a software system that clicks and types — through the same robot control task, collecting data autonomously.

The practical consequence is that organizations that cannot afford teleoperation hardware — six-axis motion-capture systems, force-sensing exoskeletons, or VR rigs — can still produce training data. Industry estimates place teleoperation facility costs at $50,000 to $150,000 per rig and output at fewer than 200 demonstrations per operator per day. GUMI requires a keyboard. The fine-tuned results also suggest that simulated demonstrations transfer to real hardware: trained exclusively on simulation data, the fine-tuned Qwen3.5-2B adapter achieves 13 of 20 successful episodes on a real robot for a task where both VLA baselines (π0.5 and GR00T N1) score 0 of 20.

Read more: TwinDEX Solves Robot Manipulation's Data Problem at the Hardware Design Stage

How Show-Harness Compares to VLA and Code-as-Policy Approaches

Purpose-built vision-language-action models represent the current mainstream approach to general robot control. Physical Intelligence's π0.5 adds open-world generalization to the original π0 flow-matching policy; NVIDIA's GR00T N1 pairs a 1.34B-parameter VLM backbone with a diffusion transformer action head, inferring 16 action steps per 63.9ms call on an L40 GPU. Both require substantial embodiment-specific pretraining data. The paper's authors report that Show-Harness in zero-shot mode achieves 89% on cross-task benchmarks against a best baseline of 57%; in cross-embodiment tests, 93% versus 52%. These are the authors' own evaluations, and independent reproduction has not yet been published.

The code-as-policy paradigm, which asks LLMs to generate Python programs that orchestrate motion primitives, has been explored since Google's original Code as Policies work in 2023. It shares VLM-based reasoning with Show-Harness but differs fundamentally in execution: the language model produces code that is then run in a separate execution environment, introducing a second point of failure (code correctness) and exposing the robot's full API to the model. Show-Harness avoids both issues by making the model responsible only for one of twelve symbolic choices per step, with the physical execution entirely inside the deterministic interpreter.

The comparison that most directly challenges Show-Harness's claims is the real-robot generalization gap. The zero-shot results are strong on the paper's own benchmarks, but the system has not yet been independently tested outside the NUS lab on tasks and environments that the research team did not design. Benchmark contamination risk is present whenever the team developing the system also constructs the evaluation — a limitation the paper does not address explicitly.

What the Remaining Limitations Reveal

The paper's own ablations are unusually transparent about where the system fails. Removing multi-view guidance — camera perspectives from multiple angles — alone drops success by 38 points. This indicates that the current semantic action vocabulary is not robust to single-view partial occlusion, a common condition in real-world deployment. The adaptive step plugin, which allows the interpreter to vary how far it translates the arm per action unit, is required to reach 96% on some tasks; without it, success falls to 70% on coarse-step and 82% on fine-step configurations, indicating the metric mapping in the interpreter must be tunable to task requirements.

The requirement for both semantic naming and directional conventions in the action vocabulary is a deployment risk: prompts that incorrectly describe "left" relative to the robot frame or inconsistently define which camera axis corresponds to which direction would likely cause systematic manipulation failures, reproducing the 5% scenario the paper documents for the unmapped case. Prompt engineering discipline becomes a load-bearing requirement.

At the inference level, the zero-shot mode's API dependency creates cost and latency exposure that the fine-tuned mode resolves locally but trades against model capability. Frontier VLMs continue to improve on spatial grounding — the identified primary bottleneck — meaning that as GPT, Gemini, and Claude models improve at 3D object localization, the zero-shot performance ceiling of this interface will rise without any changes to Show-Harness itself.

The Interface Question That Extends Beyond This Paper

The deeper research claim in Show-Harness is that deploying frontier AI on robots does not require making the AI model physical — it requires making the physical system legible to the AI model through the right interface. The team's ablation evidence supports this: performance differences between the five tested frontier VLMs correlate more strongly with spatial grounding capability than with model size or reasoning depth, and the symbolic interface itself — not the model behind it — determines whether physical control is possible at all.

That finding, if it holds under external reproduction, has implications beyond manipulation benchmarks. GUMI's extension of the same twelve-symbol space to GUI agents suggests that the semantic interface could connect frontier VLMs to any system where discrete, labeled actions are meaningful — physical or virtual. The boundary between robot control and computer-use agents begins to blur when the same action vocabulary drives both. Whether the approach generalizes to unstructured manipulation, dynamic environments, or long-horizon tasks involving dozens of subtask transitions is the question that follow-on research will need to answer. The code and models are public at the Show-Harness GitHub repository; the reproducibility test has now begun.