In one case showcased by WorldOracle, a robot's grasp missed the fruit, yet the world model generated a video in which the pick succeeded. The rollout looked plausible on screen. The success it showed never happened. Cases like this are the reason for a new world model benchmark from researchers at Princeton University, the University of Texas at Austin, the University of Toronto, NVIDIA and AgiBot.
WorldOracle evaluates action-conditioned world models, known as ACWMs. Each model receives the initial observations and the action sequence from a real manipulation episode. Its predicted rollout is then scored against the recorded execution of the same actions. The evaluation platform opened to developers on October 16 and accepts both open-weight and closed-source models.
What the seven models got wrong
The team evaluated seven representative models, including DreamDojo, Cosmos 3, IRASim and GE-Sim 2.0. They report two main problems. Robot arm motion is predicted relatively well, but the positions and movements of manipulated objects often deviate from reality. And models that do well over short horizons can degrade sharply over complete operation sequences.
To separate the two, the researchers scored the arm and the manipulated object independently in each predicted video. All five models evaluated on human-operated AgiBot G2 data scored higher on the arm than on the object. IRASim scored 78.8 on the arm and 29.1 on the object. Cosmos 3 scored 89.4 and 45.4. The other models showed gaps of varying size.
The horizon test used the same AgiBot G2 teleoperation data. WorldOracle scores each trajectory over its first 30 seconds and over the complete sequence. On the single-view score, WorldOracle-SV, all five models fell. DreamDojo dropped from 50.56 to 27.08, and Cosmos 3 from 53.99 to 26.60. GE-Sim 2.0 fell from 73.86 to 63.33, the smallest decline, and it had the highest score of the five at both horizons.
The ranking also changed. Over the first 30 seconds IRASim scored below Ctrl-World, DreamDojo and Cosmos 3. Over the full sequence it scored above all three.
How the scoring works
The benchmark starts from real manipulation records that include video and actions. Failures count as test material. If a grasp misses, a drop occurs or liquid spills in reality, a model that predicts that outcome is scored as correct. A model that defaults to a successful version of the task is not.
Coverage includes two robot embodiments, AgiBot G2 and YAM, plus first-person human-hand manipulation. Four tasks cover rigid, deformable, fluid and articulated objects: fruit picking, towel hanging, water pouring and drawer pulling. Task coverage varies by embodiment. Data come from human-operated collection and from policy rollouts in which an autonomous policy controls a real robot. Policy data can contain pauses, jitter and corrections that differ from human operation.
Four metrics are used. Physical State Correctness checks end-of-interaction outcomes such as object position, grasp state and liquid transfer. Interaction Spatial Alignment checks whether the paths of objects, arms or hands match the real process. Key Interaction Area Fidelity focuses on regions around the object, gripper and hand, combining pixel-level, perceptual and semantic similarity so that large backgrounds count for less. Multi-View Action Coherence checks alignment across synchronized camera views for models that output multiple views.
The first three metrics are averaged with equal weights to produce the WorldOracle-SV score. For multi-view evaluation, all four are weighted equally. Sub-scores are retained so developers can tell state errors from trajectory errors and local detail errors.
To validate the metrics, 119 participants compared 128 pairs of predicted videos. The four metrics agreed with human choices 88.1% to 93.2% of the time.
Tracks and submissions
There are three tracks. WorldOracle-Core covers the first 30 seconds of human-operated trajectories. WorldOracle-Policy covers complete autonomous policy episodes on AgiBot G2. WorldOracle-Long covers complete, human-operated long sequences. After registration and approval, developers pick a track, generate videos from the provided initial frames and action sequences, upload them, and receive composite scores, sub-scores and comparisons. Details are on the project page and in the paper.
What it means, and what is still unproven
The central contribution is a change in what gets measured. Many video-generation evaluations reward visual plausibility. This world model benchmark asks whether a prediction matches what a specific robot actually did, including when it failed. The missed-grasp example shows why that matters. A model that always imagines success would look fine on a plausibility test and be useless for judging a policy before running it on hardware.
The results also point to a practical gap. High arm scores alongside low object scores suggest that models are reproducing commanded motion better than they model contact and consequences. The drop over full sequences suggests errors compound as one interaction's object state becomes the next one's starting point. Both are testable weaknesses that developers can now track by name.
The authors say that, among the benchmarks they compared, WorldOracle is the only one that combines action-sequence input, real execution data, failure cases, long-horizon interaction, multiple embodiments, multi-view evaluation and policy rollouts. That is the team's own comparison, and other researchers have not yet confirmed it. The breadth is real, though, and the combination of failures and policy rollouts is the part that bears most directly on using world models to evaluate robot behavior.
Several limits remain. The benchmark compares each prediction with one recorded execution. For fluids, deformable objects and stochastic contact, the same action can plausibly lead to several outcomes, and a single reference cannot capture that range. A correct but different prediction may be penalized.
Training support is also uneven. AgiBot G2 models use a benchmark-specific training set, YAM models use public data, and the human-hand setting is zero-shot. The authors warn that scores across these settings should not be used to rank embodiment difficulty or judge cross-embodiment generalization. AgiBot is both a co-author and the source of the G2 data, so independent submissions and replication will matter for confidence in the rankings.
The metric validation is encouraging but modest. Agreement of 88.1% to 93.2% with human preference on 128 pairs shows the scores track human judgment. It does not show that a high score means a model is safe to use as a stand-in for hardware testing. That link, between benchmark score and usefulness for policy evaluation, has not been demonstrated in the material released so far, and it is the result to watch for.