SceneSmith, a system from MIT CSAIL and Toyota Research Institute, generates indoor 3D scenes with three AI agents that all run on the vision-language model GPT-5.2. The scenes load directly into physics simulation software and serve as robot training simulation environments, including restaurants, bedrooms and hotels. The team made over 1,300 scenes with it.
The work targets a data bottleneck. Robots learn best from experience, but physically teaching them many actions across many settings takes time and labor. Simulation is a natural substitute. Russ Tedrake, a principal investigator at CSAIL, says physics engines have improved significantly in recent years. The remaining challenge, he says, is creating simulation content rich and diverse enough to capture the complexity of the real world.
Three agents, one scene
Each agent has a defined role. A designer agent generates the elements of a scene. A critic agent advises whether it looks realistic and practical. An orchestrator agent manages the exchange between them and decides when the design is finished. Because each agent calls on the same vision-language model, trained on large amounts of internet text and images, each carries some spatial knowledge of how everyday places are laid out.
The process runs in stages, much like drawing a floor plan and then filling it in. The designer starts with a general layout, the critic reviews it, and the orchestrator signs off. The agents repeat the loop for furniture, then objects on walls and ceilings, and finally items a robot can manipulate. The critic might, for example, advise removing a bathtub from a living room. The orchestrator can send the design back several turns if the visuals are not good enough. Once the agents finish, simulation software adds the mechanics of the physical world.
A user can prompt the system with a request such as a garage with a car, a workbench, tires stacked in a corner and a ladder against the wall. The system can also build single objects. Asked for a rolling serving cart, it makes a 2D image and turns it into a detailed model with mass, friction and inertia. Because assets are generated from text, they are not limited to a fixed library. The scenes include articulated items such as cabinets that a robot can open and close, which the researchers say prior baselines often lacked.
What the tests showed
SceneSmith scenes hold up to up to six times more items than those from prior methods. The team compared it with the scene-generation baselines HSM and Holodeck. SceneSmith produced environments with more objects, including a private office, a pottery store and a Minecraft-themed gaming room.
More than 200 users also compared the outputs. They judged SceneSmith's visuals more realistic over 90 percent of the time. They also found that it generally followed prompts more closely than the other approaches.
Realism is hard to prove directly, so the researchers tried several angles. The most telling test used a pretrained robot policy, an AI controller trained largely on real-world data that had never seen a SceneSmith scene. Told to take an apple from a bowl and place it on a cutting board, the simulated robot did so. The researchers argue that this would not have worked if the scenes differed too much from the real settings the policy had learned from. They also teleoperated robots through the spaces to open cabinets, put away bottles and move between rooms, which they say showed the environments withstand sustained physical interaction.
The team also used the scenes to evaluate action plans. They tested different policies across 100 unique spaces, and a VLM agent judged each attempt. It found the robot's plans faulty, with the machine often failing at its chores. Humans agreed with the model's verdicts over 99 percent of the time. The researchers say this could help roboticists weed out flawed approaches before a robot moves in the real world.
Nicholas Pfaff, an MIT PhD student and lead author, says the system produced creative and diverse arrangements that he had not taught it through prompts. The scenes suit practice tasks such as putting a cup in the sink, placing fruit on plates and moving a soda can from a shelf to a table.
Limits the team acknowledges
Speed is the main cost. A single scene can take multiple hours, because the agents create and closely scrutinize each object. The researchers say more computing power could raise efficiency substantially. They also hope to extend the system to deformable objects such as sponges, if extensive 3D libraries become available.
The paper's authors are Pfaff and Tedrake, with Thomas Cohn of MIT and Toyota Research Institute roboticists Sergey Zakharov and Rick Cory. Support came in part from Amazon, the U.S. Office of Naval Research, Toyota Research Institute and the U.S. National Science Foundation. The team presented the work as a spotlight at the 2026 International Conference on Machine Learning.
Analysis: what this changes and what it does not
The practical value is in scale and variety. Hand-built simulation scenes are costly to author, and a text prompt that yields a populated, physics-ready room moves that cost from human designers to compute. The reported object density and the inclusion of articulated, manipulable items matter because manipulation practice needs things to open, grasp and move. By those measures, and against HSM and Holodeck, SceneSmith is a step forward in content richness.
Several questions remain open. The strongest evidence of realism is a pretrained policy completing a task in the generated scenes. The source describes one example, the apple and cutting board, and does not give a success rate across scenes or tasks. That supports the claim that the scenes resemble real settings, but it does not show that a policy trained inside them performs better on physical hardware. The source does not report that test.
The evaluation loop also leans on a VLM. Humans agreed with the model's verdicts over 99 percent of the time, which is a strong result. It was measured on 100 spaces and a particular set of policies, so it should be treated as a check on that setup rather than a general guarantee. The whole pipeline also depends on one model, GPT-5.2. How well it works with other models is not reported.
Throughput is the other constraint. Multiple hours per scene is acceptable for building a library of environments in advance, but it limits any workflow that needs scenes generated on demand. The exclusion of deformable objects also leaves out common kitchen and household items. The reported gains come from richer content, and none of the results show that this robot training simulation approach has yet cut real-world testing in practice, though the policy-screening results point in that direction.