When Ai2 launched Molmo they bet on something concrete: openness. Not just open weights and code, but models you can inspect, adapt and replicate. That bet now turns into an ecosystem that not only sees, but also points and acts in the digital and physical world.
MolmoPoint: more efficient cross-modal pointing
Pointing sounds simple, right? But for a vision-and-language model (VLM) doing it well is surprisingly hard. The classic approach turns an X,Y coordinate into text — an indirect, brittle shortcut — and demands very specific training mixes and lots of data.
MolmoPoint changes the game: instead of generating coordinates as text, the model directly selects what it already sees. First it picks a coarse region, then refines down to the exact point. It’s a cross-modal solution: the same mechanism can point in images, video sequences, or even at fragments of other inputs like text or audio.
What do you get? Big gains in training efficiency and benchmark metrics: better pointing performance, improved detection of elements on screens, and stronger object tracking compared to similar open models. It works especially well at high resolution and on interfaces full of tiny buttons. MolmoPoint ships with variants for images, video and UIs, plus open datasets with thousands of annotated screenshots and human tracks.
Teaching a VLM to point no longer requires dedicating a large slice of the training mix only to pointing data; this lowers cost and speeds up future training.
MolmoWeb: visual agents that interact with the web
MolmoWeb is a suite of multimodal agents that, given a screenshot and an instruction, predict the browser’s next action. They don’t use the DOM or accessibility trees: they perceive the interface like a person and manipulate it with mouse and keyboard.
Why work with screenshots? They’re more robust to site changes and cheaper to process: one image can represent thousands of lines of code. The architecture prioritizes visual perception and sequential action, and it’s trained with a winning mix: synthetic data generated by LLMs, human demonstrations (the largest public collection reported), and supervised fine-tuning.
The challenges aren’t trivial: evaluating agents is expensive because a failure at one step can break the whole trajectory. That’s why Ai2 built an eval harness and trajectory-visualization tools to spot inconsistencies between data generation, training and evaluation.
MolmoWeb outperforms comparable open-weight models and, in its most capable configuration, matches or exceeds agents based on much larger proprietary models like GPT-4o — despite having fewer parameters and training only on screenshots.
Open building blocks: from robotics to AR
MolmoPoint and MolmoWeb share the same Molmo 2 visual backbone and the openness philosophy. Together with MolmoBot, MolmoSpaces and WildDet3D, the ecosystem covers pointing, web interaction, 3D perception and physical manipulation.
That has practical consequences: a university can fine-tune MolmoPoint for tracking animal behavior; a team in Edinburgh already used it in multimodal debate frameworks; another group used the open pipeline to study how VLMs understand spatial relationships. Openness enables reproducibility and accelerates research.
Technical and practical implications
- Architecture and training: shared visual backbone, supervised fine-tuning with mixes of synthetic and human data, and a pointing approach based on direct selection instead of textual output.
- Data and reproducibility: checkpoints, code, datasets and evaluation harnesses are public so others can reproduce and improve results.
- Performance: gains in training efficiency and in high-detail tasks (dense UIs, video tracking) compared to prior generations and similar open models.
Technically, this means less dependence on finely tuned training mixes and bigger wins when you integrate practical tasks (e.g., web automation or robotic manipulation) from dataset design through to evaluation metrics.
Why it matters for you
Because these technologies aren’t only for big labs. A reliable visual agent can automate repetitive web tasks, help people with digital barriers navigate complex sites, or let a local robot identify where to grasp an object with greater precision. And since everything is open, the spark of innovation can come from a startup, a small university, or a hobbyist trying to solve a concrete problem.
Molmo is no longer just a model; it’s a collection of open tools designed so the community can improve how machines see and act. What project could you build with these open blocks?