RoboGen: Towards Unleashing Infinite Data for Automated Robot Learning via Generative Simulation

Scaling up robotic skill acquisition has historically faced severe bottlenecks due to the slow, risky nature of real-world trials and the labor-intensive requirements of building simulation environments. Traditionally, human engineers must manually design virtual environments, handcraft three-dimensional assets, formulate scene layouts, and construct mathematical reward functions for every new capability. The article introduces RoboGen, an automated agent designed to eliminate these manual constraints by establishing a self-guided propose-generate-learn framework for automated robot learning in simulation.

The primary objective of the article is to demonstrate that foundation and generative models can autonomously configure tasks, environments, and training supervisions to scale robotic skill acquisition with minimal human intervention. Rather than directly using large language models to control robot joints—an approach that often struggles due to a lack of physical grounding—the system uses language and vision models for task imagination, spatial reasoning, and supervision design, while delegating low-level physics execution to physics-grounded simulators.

The authors implemented this approach using a self-guided propose-generate-learn pipeline operating on the differentiable simulation platform Genesis, with OpenAI's GPT-4 as the primary language backend and Gemini-Pro as a visual verifier. The system queries the language model to generate tasks conditioned on robot capabilities, creates environments by retrieving meshes from the Objaverse database or generating them via image-to-3D pipelines, automatically verifies realistic object scales, decomposes tasks into sub-tasks, and selects the most appropriate learning algorithm. RoboGen chooses between reinforcement learning, gradient-based trajectory optimization, and motion planning depending on whether the task involves articulated manipulation, deformable soft-body interaction, or legged locomotion.

Evaluation showed that RoboGen achieved superior task and visual diversity compared to established human-curated robotic benchmarks (such as RLBench, ManiSkill2, Meta-World, and Behavior-100) and concurrent procedural generation systems, as measured by lower text redundancy and image similarity scores. Across a benchmark suite of 69 diverse tasks spanning articulated objects, soft materials, and quadruped locomotion, the automated learning pipeline achieved an overall training success rate of 77.4%. Hybrid algorithm selection proved essential; relying solely on reinforcement learning for articulated manipulation caused nearly all tasks to fail, whereas pairing motion planning with learning enabled reliable execution. Furthermore, automated visual and dimensional verification significantly reduced asset failures, and an analysis of 155 generated tasks identified only 19 failure cases, primarily caused by asset geometry mismatches or language ambiguities in object joint limits.

These findings demonstrate that generative simulation offers a scalable, cost-effective path to producing massive volumes of diverse robotic training data without requiring continuous manual engineering. Organizations developing autonomous systems can substantially reduce development cycles and human overhead by automating task design and reward authoring. However, real-world deployment remains bounded by the simulation-to-reality gap, requiring techniques such as domain randomization to transfer policies safely. Next steps include integrating multimodal environmental feedback loops to autonomously verify learned skills and self-correct reward errors, as well as conducting physical real-robot transfer trials before broad deployment in production environments.