MolmoBot: robotic manipulation trained entirely in simulation | Keryc
MolmoBot proposes a provocative but practical idea: what if you could train robots that can manipulate real objects without touching a single physical robot during training? AllenAI releases a full suite trained exclusively in simulation that achieves zero-shot transfer to real robots, and sparks an important conversation about how we scale robotics now that perception and reasoning have come so far.
What MolmoBot is and why it matters
MolmoBot is a suite of robotic manipulation policies trained entirely with synthetic data. It's not just a model: it's the whole stack. AllenAI publishes the training data, the tools to generate it (MolmoSpaces), the training code, and a technical report so others can reproduce and extend the work.
Why is this a game changer? Because the biggest practical bottleneck in robotics has been collecting costly, manual real-world data. Projects like Open X-Embodiment and DROID show the scale of that problem: millions of trajectories or hundreds of hours of teleoperation. MolmoBot proposes shifting the bottleneck to designing better virtual worlds—something that scales with compute and open access.
How they trained everything in simulation
The core recipe mixes three concrete ingredients:
MolmoSpaces: an open platform for procedurally generating environments and trajectories.
MolmoBot-Data: millions of expert synthetic trajectories generated with MuJoCo, heavy domain randomization, and active variations in objects, lighting, textures, cameras, and dynamics.
Training from RGB observations only and behavior cloning on those trajectories.
The strategy is ambitious: instead of using simulation as support, they make it the sole data source. To close the sim-to-real gap they bet on extreme diversity in scenarios and virtual sensors: fully randomized cameras, object models taken from iTHOR and Objaverse, and aggressive physical variations.
Relevant technical details
Physics engine: MuJoCo for realistic contact and manipulation simulation.
Signals used: during training they can generate depth and privileged metadata, but the policies learn only from RGB, which makes the transfer more notable.
Supervision: behavior cloning at scale, with no reinforcement learning or real-world fine-tuning.
Architectures and tasks
MolmoBot isn't a single network. It's a family that covers different trade-offs of capacity and compute:
MolmoBot: the main vision-language policy built on the Molmo2 backbone. It processes multiple timesteps of RGB and natural language instructions and achieves the best metrics.
MolmoBot-SPOC: a lightweight variant adapted from the SPOC design, parameter-efficient and useful where compute is limited.
MolmoBot-Pi0: uses the PaliGemma backbone with an action head to compare directly with the π family of Physical Intelligence.
Tasks evaluated (on two real platforms: Rainbow RB-Y1 and Franka FR3):
Pick-and-place (Franka FR3).
Manipulation of articulated objects: drawers, microwaves, doors (RB-Y1).
Door opening and mobile manipulation (RB-Y1).
Tasks can be specified in natural language or with simple commands like "pick", "place" or "close".
Results: zero-shot sim-to-real and comparisons
With no real-world tuning, MolmoBot transfers to both robots and to objects and scenes unseen during training. In pick-and-place it outperforms π0.5 (a model trained with large-scale real data) and records competitive performance with π0 under standardized protocols.
They also tested robustness to unseen visual changes: camera variants, lighting shifts, and even a different renderer at evaluation. Those tests show that scale and synthetic diversity can compensate for the lack of real data in many manipulation tasks.
Limitations and open questions
Not everything is solved. A few limitations to keep in mind:
Scope of transfer: the suite shows that many everyday manipulation tasks are reachable, but it doesn't guarantee that every real-world condition is covered. Edge cases and failures remain instructive.
Complex physics and contacts: MuJoCo is powerful, but certain fine interactions with deformable materials or emergent behaviors may need better physics models or real validation.
Task definitions and metrics: paper-to-paper comparisons are sensitive to success criteria and setup details; AllenAI tries to match protocols, but it isn't always trivial.
Does this mean real data collection disappears? Not entirely. It means its role shifts: instead of being the sole supervision source, real data can be used to timestamp digital twins, validate, and close specific gaps.
What changes for researchers and entrepreneurs
Democratization: labs with fewer resources can experiment with real manipulation without investing in many hours of teleoperation.
Faster iteration: generating virtual worlds and retraining policies is much cheaper than deploying robot fleets and collecting data.
Reproducible research: publishing synthetic data, pipelines, and code makes it easier to replicate and compare methods.
If you work on manipulation, sim-to-real, or instruction grounding in the physical world, MolmoBot gives you tools ready to try on your robot or benchmark. The authors explicitly invite others to find the weak points: those failures will guide the next generation.
MolmoBot isn't the final word, but it's a strong proof that simulation, done at scale and with diversity, can shift the main burden of data acquisition. The conversation now moves to how we design rich, varied virtual worlds and how much we can trust policies trained only in simulation.
