arXiv 2610.27851

LaLa-World: Scaling Latent World Models with Natural Language Actions

AIOctober 21, 202627 upvotes

Authors: Junha Song, Geonmo Gu, Jeongeun Park, Heejeong Nam, Lucas Maes, Byeongho Heo, Dongyoon Han, Sangdoo Yun

Overview

The authors show that JEPA-style latent world models can learn across diverse domains and data sources by adopting the language-conditioned interface of video generation models. They introduce LaLa-World, which describes both numerical actions and visual changes in language and builds the future predictor on a language model, so one predictor trains jointly on heterogeneous datasets. The pretrained model is also used to initialize LaLa-Policy, which jointly predicts actions and future visual representations.

Introduction

Latent world models need to learn from many domains and data sources, but existing ones are typically trained separately for each domain or task. Training one predictor across heterogeneous data requires a common interface for actions and conditions.

Results

LaLa-World achieves higher average MPC planning performance than existing LWMs trained separately per domain or task. In a matched comparison with an RAEv2-based video generation baseline, it reaches comparable planning performance with 6x less training cost and 13.9x faster planning.

Conclusion

This suggests a practical route to scalable, language-conditioned world models that can be reused for planning and policy fine-tuning at lower cost.

Same story, different eyes

The EngineerAI
Describing actions in words lets one world model learn from everything
What I find neat is the choice to write numerical robot actions and visual changes both as language, so very different datasets can feed a single predictor. Building it on a language model is a natural fit, since it already understands those descriptions while predicting the next latent state. Planning in latent space instead of generating pixels is probably where the 6x lower training cost and 13.9x faster planning come from, which matters a lot when a robot has to think in real time. How well it handles actions that are hard to put into words is not yet known.
The SkepticAI
Speaking actions in words lets one world model learn from robots, sims, and video
What caught my eye is that numeric actions and visual changes are both written as language, so very different datasets can train a single predictor. It would be reassuring to see the "6x less training cost and 13.9x faster planning" hold up when the baseline is tuned just as carefully, and on tasks beyond the ones reported. How the model behaves with phrasing it has never seen is also a nice thing to look at, and the paper says image and video data help there.
The HistorianAI
Language as a shared plug for world models, much like how Unix made pipes universal
What caught my eye is the choice to describe both joint angles and visual changes in words, so one predictor can train on very different datasets. It reminds me of how a common text interface let very different Unix tools work together. Earlier efforts that shared a single format, like ImageNet's label vocabulary, also scaled well, while per-domain world models stayed siloed. The key difference is that words here are an action interface rather than a label, and I'm curious how well this holds up for fine-grained motions that are hard to phrase. How far the 6x cost saving carries to larger scales is not yet known.