As AI agents become increasingly capable, agent-driven robotic control is emerging as a compelling paradigm. However, prevailing vision-language-action (VLA) models and world action models (WAMs) still rely on natural-language instructions to specify manipulation tasks, an ill-suited interface for agent-driven control: referentially ambiguous, spatially imprecise, redundant with the agent’s inherent language understanding, and entangling intent with execution.
We present AR-WAM, a visual-conditioned, agent-ready world action model that replaces language with two complementary conditions: a visual grounding prompt (a bounding box of the target) denoting the interaction object and location, and a learnable operation token dictating the atomic skill to execute. Our compact 0.5B-parameter model, with a frozen pretrained visual encoder and no language encoder, predicts scene evolution within compact latent states while decoding actions, exposing the policy’s intent through explicit, supervisable reasoning signals. A model-agnostic compatibility layer provides three primitives (detect, execute, and query) so that local VLMs or online agent APIs can drive the policy directly, with long-horizon memory and closed-loop error recovery delegated to the agent side.
On RoboTwin 2.0, RMBench, and a real Astribot S1 dual-arm platform, AR-WAM matches the strongest baselines on standard manipulation (87.2% average success) and outperforms them on memory-dependent and real-robot long-horizon tasks, improving success rates by 5.9% and 36.7%, respectively, while maintaining the lowest inference latency (14.1 ms).
An agent grounds the target and issues a visual prompt g together with an atomic operation s. A compact 0.5B world action model (a 0.2B transformer shared by a reasoning expert and a world action expert, plus a frozen 0.3B vision encoder) predicts an action chunk together with dual-level foresight signals. The reasoning expert runs once per control cycle, and the world action expert then performs multi-step denoising while reusing its KV cache.
A bounding-box visual grounding prompt specifies where to act, and a learnable operation token specifies which atomic skill to perform (pick, place, click, handover, …). Spatial conditioning replaces text: unambiguous, precise, and free of redundant language understanding inside the policy.
A reasoning expert externalizes task intent as explicit, supervisable signals (target mask, end-effector poses, and a task abstract), while visual and semantic foresight predictions keep action generation future-aware. Both experts share one compact transformer.
Three model-agnostic primitives (detect, execute, and query) let any VLM/LLM agent drive the policy directly. Long-horizon memory and closed-loop failure recovery are delegated to the agent, keeping the policy Markovian and compact.
Future-aware action generation in compact latent form
A reasoning expert runs once per control cycle and externalizes task intent as explicit, supervisable signals: a target mask, end-effector poses, and a task abstract. On top of it, dual-level foresight keeps action generation future-aware: an offline foresight-gist extractor compresses the temporal evolution between two endpoint frames into a compact latent gist, with no actions or intermediate frames, and the policy is aligned to predict both the visual foresight of the future scene and the semantic foresight of the target within it. Removing foresight alignment costs 8.6% average success, and replacing the compact gist with raw future patch features drops 10.8%, below even the foresight-free variant. Reasoning attends to the object and the target; foresight attends to where motion will occur.
Current RGB, foresight RGB, and reasoning masks for the current and foreseen scene, on RoboTwin (Handover Block) and RMBench (Cover Blocks).
Event-triggered agent invocation through three primitives
The agent is invoked only on trigger events: end-effector displacement, gripper open/close, or a timeout. It grounds the target with its own perception (detect), rolls out the policy until the next event (execute), and receives structured feedback for its next reasoning step (query). Commands and feedback accumulate in the agent’s history, so long-horizon memory lives entirely on the agent side. On a grasp failure or a dropped object, the agent re-localizes the target and retries under a fixed budget, and the policy’s decoded intent helps diagnose whether a failure stems from wrong intent or improper execution.
Failure recovery through the agent-policy loop: the agent detects the failure, re-issues an updated visual prompt, and retries, with no hardwired recovery behaviors.
Recovery rollouts on RoboTwin 2.0 (×5 speed): after a failed grasp, the agent re-grounds the target and issues a fresh visual prompt to retry.
RoboTwin 2.0 and RMBench in simulation, plus a real Astribot S1 dual-arm platform
Ten dual-arm long-horizon tasks composing rich combinations of atomic skills, with instructions issued by a benchmark expert equipped with an object detector. AR-WAM attains the best average success rate in the randomized setting (84.2%) and stays within 0.6% of the strongest baseline in the clean setting (87.2% vs. 87.8%), with visual prompts instead of language instructions. Replacing the visual prompt with text under identical training drops average success by 13.0%.
| Task | π0.5 | X-VLA | Motus | Fast-WAM | LiLa-WAM | AR-WAM (Ours) | ||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| C | R | C | R | C | R | C | R | C | R | C | R | |
| Blocks Ranking Size | 49 | 26 | 67 | 74 | 75 | 63 | 94 | 98 | 92 | 88 | 100 | 94 |
| Handover Block | 66 | 57 | 73 | 37 | 86 | 73 | 99 | 80 | 96 | 90 | 98 | 92 |
| Handover Mic | 98 | 97 | 0 | 0 | 78 | 63 | 100 | 100 | 100 | 98 | 100 | 100 |
| Hanging Mug | 18 | 17 | 23 | 27 | 38 | 38 | 65 | 56 | 56 | 44 | 54 | 62 |
| Place Can Basket | 62 | 62 | 49 | 52 | 81 | 76 | 72 | 67 | 80 | 72 | 90 | 82 |
| Place Dual Shoes | 75 | 75 | 79 | 88 | 93 | 87 | 88 | 88 | 60 | 54 | 86 | 78 |
| Put Bottles Dustbin | 84 | 79 | 74 | 77 | 81 | 79 | 93 | 82 | 92 | 94 | 96 | 92 |
| Put Object Cabinet | 80 | 79 | 46 | 48 | 88 | 71 | 94 | 82 | 92 | 92 | 78 | 70 |
| Scan Object | 72 | 65 | 14 | 36 | 67 | 66 | 96 | 86 | 94 | 90 | 76 | 86 |
| Stack Bowls Three | 77 | 71 | 76 | 86 | 79 | 87 | 77 | 86 | 88 | 82 | 94 | 86 |
| Average | 68.1 | 62.8 | 50.1 | 52.5 | 76.6 | 70.3 | 87.8 | 82.5 | 85.0 | 80.4 | 87.2 | 84.2 |
Success rates (%) over 50 trials per task, per setting. C: clean setting; R: randomized setting. Bold: best; underlined: second best.
Rollouts on all ten tasks (clean vs. randomized), with the agent-issued bounding box and the operation token shown alongside the policy’s reasoning readout.
Memory-dependent tasks that test the cooperation between the policy and the agent, driven by off-the-shelf agents (a locally deployed Qwen3.8-27B, AR-Q, and the online API Kimi-K3, AR-K) through the model-agnostic interface. On the memory-heavy M(n) tasks, AR-WAM reaches 52.5% / 63.5% average success vs. 38.7% for the strongest memory-augmented baseline, thanks to cleanly delegating memory management to the upstream agent.
| Task | π0.5 | Fast-WAM | Goal2Skill | Mem-0 | Cortex | LiLa-WAM | AR-K | AR-Q |
|---|---|---|---|---|---|---|---|---|
| Observe and Pick Up | 9 | 0 | 8 | 4 | 14 | 10 | 44 | 84 |
| Rearrange Blocks | 13 | 0 | 38 | 89 | 100 | 20 | 62 | 86 |
| Put Back Block | 11 | 0 | – | 90 | 100 | 14 | 58 | 88 |
| Swap Blocks | 24 | 0 | – | 67 | 99 | 10 | 68 | 82 |
| Swap T | 15 | 7 | – | 14 | 63 | 18 | 18 | 16 |
| M(1) Tasks Average | 14.4 | 1.4 | 23.0 | 52.8 | 75.2 | 14.4 | 50.0 | 71.2 |
| Battery Try | 16 | 20 | 46 | 28 | 37 | 24 | 38 | 52 |
| Blocks Ranking Try | 6 | 26 | 60 | 18 | – | 30 | 18 | 34 |
| Cover Blocks | 0 | 0 | – | 68 | – | 0 | 72 | 78 |
| Press Button | 0 | 0 | 10 | 0 | 20 | 0 | 82 | 90 |
| M(n) Tasks Average | 5.5 | 11.5 | 38.7 | 28.5 | 28.5 | 13.5 | 52.5 | 63.5 |
| Total Average | 10.4 | 5.9 | 32.4 | 42.0 | 61.9 | 14.0 | 51.1 | 67.8 |
Success rates (%) over 50 trials per task. AR-K / AR-Q: AR-WAM integrated with Kimi-K3 / Qwen3.8-27B as the agent. –: not applicable / not reported. Bold: best; underlined: second best.
The agent-policy closed loop on memory-dependent tasks: the agent reasons over its history and issues a bounding box plus an atomic skill; the policy executes and reports back.
Three challenging long-horizon tasks on a real Astribot S1 dual-arm platform, all requiring agent-side reasoning and memory: pressing bells a specified number of times, picking up a specified bottle twice, and placing an egg into a specified cell. For each task, the simulation-trained policy is fine-tuned on 100 real-world demonstrations and driven by a locally deployed Qwen3.8-27B. AR-WAM achieves the highest success on all three tasks (71.7% average vs. 35.0% for π0.5).
| Task | π0.5 | LiLa-WAM | AR-WAM (Ours) |
|---|---|---|---|
| Press bells X times | 5 | 10 | 70 |
| Pick up X bottle twice | 45 | 40 | 65 |
| Place egg into X cell | 55 | 45 | 80 |
| Average | 35.0 | 31.7 | 71.7 |
Success rates (%) over 20 trials per task. Bold: best.
Real-robot rollouts of the three tasks (×5 speed). The red box is the agent-issued visual prompt.
Measured on a single RTX 4090 in bf16 with identical inputs. AR-WAM attains the lowest end-to-end latency (14.09 ms per action chunk), 4.4× faster than π0.5 and 22.7× faster than Fast-WAM, benefiting from the shared reasoning-denoising forward pass and KV-cache reuse.
| Method | Encoder & Misc. | KV Cache | ODE (per step) | End-to-end |
|---|---|---|---|---|
| π0.5 | 19.74 | 23.36 | 2.11 (×9) | 62.09 |
| Fast-WAM | 27.55 | 52.56 | 26.58 (×9) | 319.33 |
| LiLa-WAM | 4.77 | – | 1.02 (×10) | 14.97 |
| AR-WAM (Ours) | 4.41 | 3.74 | 0.66 (×9) | 14.09 |
Per-stage inference latency (ms). ODE: latency per denoising step, with the number of steps in parentheses. Bold: fastest; underlined: second fastest.
@article{jiang2026arwam,
title={AR-WAM: A Visual-Conditioned Agent-Ready World Action Model for Robotic Manipulation},
author={Jiang, Yicheng and Gan, Zesen and Wang, Xiaobo and He, Tianlun and Zhao, Chenxu and Wu, Minghui and Wang, Xinyue and Wang, Jiaxu and He, Junhao and Wang, Jianan and Shao, Qiming},
journal={arXiv preprint arXiv:2609.23578},
year={2026}
}