AI Insight
This paper introduces Orbis 2, a hierarchical world model for autonomous driving that operates at two levels of abstraction: a high-level predictor for coarse scene structure over long time periods and a low-level generator for detailed predictions. The researchers propose a two-stage training approach that first pretrains with diffusion forcing to learn rich representations, then fine-tunes with teacher forcing for stable predictions. The model achieves state-of-the-art performance on driving benchmarks including long-horizon video generation, steering response in counterfactual scenarios, and internal representation quality.
Why it matters
This work addresses a critical limitation in autonomous driving systems by combining high-quality visual prediction with spatial and semantic understanding necessary for real-world decision-making. The hierarchical approach and novel training paradigm could improve the safety and reliability of autonomous vehicle planning systems.
Understand the Science
arXiv:2607.15898v1 Announce Type: cross
Abstract: Current world models operate at a single level of abstraction, with most prioritizing perceptual fidelity while lacking the spatial reasoning and semantic understanding required for real-world downstream tasks. We present a hierarchical driving world model that factorizes future prediction across two levels operating at distinct temporal and abstraction scales: a high-level predictor that forecasts coarse scene structure over extended temporal horizons, and a low-level generator that produces detailed predictions conditioned on the high-level output. This decomposition yields high perceptual fidelity while also capturing strong spatial and semantic representations. We further show that pretraining with a diffusion forcing objective yields substantially richer internal representations than the standard teacher forcing objective, while teacher forcing — predicting only the next frame from clean context — produces more stable autoregressive rollouts. We therefore introduce a generic two-stage training paradigm that pretrains the model with diffusion forcing and fine-tunes with teacher forcing, combining the representational benefits of the former with the rollout stability of the latter. Our approach achieves state-of-the-art results across the standard suite of driving world model evaluations on established benchmarks, including long-horizon generation fidelity, steering responsiveness evaluated on counterfactual scenarios, and internal representation quality. Project page with code, demo, checkpoints and qualitative results: https://lmb-freiburg.github.io/orbis2.github.io/