LeRobot v0.6.0 launches world models and unified evaluation | Keryc
LeRobot v0.6.0 arrives with a clear play: imagine the future to improve policies, measure when they work, and make all of that reproducible and fast. Sounds ambitious, right? Does it work? This release gives you concrete tools to explore the question — from policies that learn to predict their own rollouts to a rewards API and six new benchmarks ready to run.
Modelos de mundo: políticas que imaginan
Why imagine? Because letting a policy predict what’s coming can give powerful supervisory signals during training. v0.6.0 brings three distinct approaches, each optimized for cost and performance.
VLA-JEPA: trains a compact VLA (Qwen3-VL-2B base) to anticipate frames in latent space via a JEPA. The trick: the world model supervises during training and disappears at inference, so you don’t pay extra cost when running the policy. There are ready checkpoints on the Hub and a direct fine-tuning flow.
LingBot-VA: an autoregressive video-action model that predicts video and actions in blocks, reinjecting real observations to avoid drift. You can save the imagined videos and compare them with reality; inference runs on a single GPU (24–32 GB).
FastWAM: combines an expert video generator (~5B) with a compact action expert into one network. It learns to “dream” its rollouts during training, but skips that generation at inference and denoises actions directly. Designed so the "dream" improves the policy without penalizing production latency.
VLAs: a growing zoo with technical intent
v0.6.0 adds several new families, from large to very compact, each designed for different GPU budgets and research goals.
GR00T N1.7: updated integration, now with Cosmos-Reason2-2B (Qwen3-VL) and parity with the original NVIDIA implementation. Installation is easier and flash-attention is optional.
MolmoAct2: fully ported; supports fine-tuning (full or LoRA), evaluation and deployment on real robots. It needs little VRAM at inference (about 12 GB bf16) and LoRA fits in 24 GB.
EO-1: Qwen2.5-VL-3B with a flow-matching action head, contributed by the paper authors.
Multitask DiT, EVO1, and a Multitask Diffusion Transformer policy: models ranging from ~450M to 1B, with diffusion or flow-matching objectives, ideal for training on accessible machines.
The technical takeaway? You don’t always need giant models; there are trade-offs between size, latency, and generalization.
Reward models: knowing when your robot succeeds
The new lerobot.rewards API unifies access to reward models. Two key additions:
Robometer: a generalist model trained on comparisons of trajectories from over a million episodes. Based on Qwen3-VL-4B, it scores progress and success from video and a language instruction.
TOPReward: a zero-shot approach that evaluates the probability of the token "True" given the video and instruction, using a VLM as a reward function.
Both include labeling scripts that produce per-frame progress curves, ready for RA-BC, dataset quality inspection, and visual overlays.
Datasets: depth, automatic annotations and faster loading
Codec and encoding: you can now choose codec, quality, pixel format, GOP and look for hardware accelerators like NVENC or VAAPI. Re-encoding a dataset is straightforward with lerobot-edit-dataset.
Depth support: record depth maps in compressed 12-bit, stored alongside RGB and decoded to physical units at training time.
Automatic annotations: lerobot-annotate uses a VLM to produce timestamped subtasks, plans, memory and QAs per camera. This generates YAML recipes that render as chat-style messages when sampled — useful for long-horizon policies that talk and act.
Speed: data loading up to 2x faster via parallel decoding, compact uint8 frames between workers and persistent caches. Deterministic, resumable sampling to restart training exactly where you left off.
Benchmarks: one CLI to rule them all and six new sims
Six simulation benchmarks join the ecosystem, all runnable with lerobot-eval and with ready Docker images:
LIBERO-plus: stress with 10,000 perturbed variants.
RoboTwin 2.0: 50 bimanual tasks in SAPIEN, with 100k trajectories ready.
RoboCasa365: 365 cooking tasks across 2,500 procedural kitchens.
RoboCerebra: long-horizon episodes with 3–6 sub-goals.
RoboMME: memory exams and tracking of hidden objects.
VLABench: reasoning and manipulation knowledge.
Evaluation is faster thanks to asynchronous vectorized environments, and each benchmark brings its dependency recipe or a Docker image to avoid install pain.
Deployment, DAgger and human-in-the-loop collection
The new lerobot-rollout separates deployment from logging. It includes strategies: base, sentry, highlight, episodic and dagger. DAgger enables real-time human corrections: you take control, record the intervention and the dataset is ready for fine-tuning. That turns the deploy-collect-correct-train cycle into a reproducible flow with a single flag.
FSDP with Accelerate: shards parameters, gradients and optimizer state so you can train big models across many GPUs and gather checkpoints into a single model.safetensors that’s easy to load.
HF Jobs: the same lerobot-train command can run in the cloud with --job.target, uploading your dataset if needed and returning the model to the Hub. From T4 to H200, pay-as-you-go.
Practical mixed precision: --policy.dtype=bfloat16 enables bfloat16 training via Accelerate.
Code, installation and user experience
Lighter install: about 40% fewer base dependencies and feature extras. Error messages tell you exactly which extra is missing.
Compatibility: PyTorch 2.7–2.11, CUDA 12.8 wheels targeted for Linux. Pip-installable plugins auto-register.
UX: Foxglove as a remote display, keyboard controls on Wayland and macOS without extra permissions, and a UI called LeLab for the full flow if you’d rather avoid the CLI.
Community and ecosystem
v0.6.0 includes contributions from academia, industry and hobbyists. New guides cover everything from adding a benchmark to publishing a policy plugin without a PR. There’s a compute guide that tells you which GPU you need and how long things will take, with measured times from an RTX 4090 up to 4x H100.
The release also brings hundreds of fixes and docs improvements; it’s built for reproducible research and to make the jump from lab to deployment easier.
Think of this as a toolbox: you now have policies that imagine, automatic ways to measure success, pipelines to enrich data with language, and a set of benchmarks to test hypotheses. If you want to experiment with world models in robotics, v0.6.0 gives you the technical and operational foundation to do it at scale.
