MolmoAct: Open AI that reasons in 3D space | Keryc
Can you imagine an AI that thinks in three dimensions the way you and I think with maps and sketches? MolmoAct is exactly that: a model that combines perception, visual planning and control to reason about actions in 3D space and execute commands on robotic hardware. This initiative was presented by the Allen Institute for AI on August 12, 2025 and comes with open code, models and data so anyone can test and adapt it. (allenai.org, ar5iv.org)
What is MolmoAct and why it matters
MolmoAct belongs to a new class called Action Reasoning Models (ARMs). Instead of turning instructions directly into motion, MolmoAct follows three chained stages: first it creates perception tokens that encode depth and position information; then it generates visual waypoints as an intermediate plan; and finally it decodes those waypoints into low-level commands for actuators. That separation makes the reasoning more interpretable and easier to transfer between different robots. (, )
Why does this matter for you? Many current models reason mostly with text and stumble when you need to estimate distances, avoid collisions, or predict object dynamics. MolmoAct converts perception into editable visual strokes, which makes it simpler for a human to correct or guide the plan before commanding the robot. (allenai.org)
What's in the release
This release is not just a paper: the Allen Institute publishes MolmoAct-7B (the initial version) along with weights, checkpoints, the training dataset and evaluation tools. The MolmoAct Dataset contains around 10,000 robotic episodes in household scenarios and is publicly available so you can reproduce and improve the results. The repositories and packages are on GitHub and Hugging Face under the Apache 2.0 license. (ar5iv.org, github.com, huggingface.co)
If you're a developer or researcher, this means you can download models and data, reproduce evaluations in simulators like SimplerEnv, and fine-tune MolmoAct for your own robotic arm or humanoid. Want to try it quickly? AllenAI published checkpoints on Hugging Face and a repo with instructions for evaluation. (huggingface.co, github.com)
Performance and efficiency
MolmoAct-7B was pretrained with a specialized data mix and mid-trained on its own dataset. The authors report efficient training: pretraining using millions of examples on clusters with H100 GPUs completed in surprisingly short schedules compared to some competitors. In simulation benchmarks like SimplerEnv and LIBERO, MolmoAct reaches success rates that outperform several major models, showing good out-of-distribution generalization. (allenai.org, ar5iv.org)
In practical terms, that means with moderate resources and the open dataset, small teams can replicate or adapt complex behaviors without relying exclusively on closed models or giant infrastructures. Isn't that exactly what startups and academic labs need to move forward? (ar5iv.org)
Control, interpretability and safety
One of the most useful features is the visual trace of reasoning: MolmoAct overlays planned trajectories on the image before executing real actions. That allows early human intervention and reduces risk in physical tests. The model also accepts manual annotations (for example, drawing a route on the screen) to guide behavior in real time. These options make audits and adjustments easier before moving real hardware. (allenai.org, github.com)
What it means for entrepreneurs and creators
If you work in automation, service robotics, light manufacturing or home robots, MolmoAct offers an open, practical foundation to:
Test new control policies using perception tokens and visual waypoints.
Speed up prototypes with checkpoints available on Hugging Face.
Avoid starting from scratch because the dataset and code let you reproduce training and evaluations. (huggingface.co, github.com)
Quick steps to get started:
Read the blog and paper to understand the architecture and benchmarks. (allenai.org, ar5iv.org)
Download the model and dataset from Hugging Face and clone the GitHub repo to reproduce the scripts. (huggingface.co, github.com)
Run the evaluations in SimplerEnv and then adapt with fine-tuning to your real robot following the safety guides. (github.com)
Technical paper and detailed metrics on arXiv. [See arXiv]. (ar5iv.org)
Models and datasets on Hugging Face. [Explore on Hugging Face]. (huggingface.co)
Official repository with license and scripts. [View on GitHub]. (github.com)
MolmoAct is not just a technical advance: it represents a paradigm shift toward models that integrate spatial intuition into the decision-making pipeline. Ready to try an AI that thinks with space and visual strokes instead of just words? Try the model and tell me which experiments you'd like to see replicated in your context.
Stay up to date!
Get AI news, tool launches, and innovative products straight to your inbox. Everything clear and useful.
