AI Insight
Researchers compared depth perception capabilities in a vision-language model (VLM) before and after training it for action prediction tasks. They found that the action-trained model (VLA) showed degraded depth understanding across all layers, with particularly severe deterioration in the final layers—a pattern they termed the "cliff." Through systematic ablation studies, they identified that this late-layer collapse is specifically caused by interference from the multilayer perceptron (MLP) components in the final layers of the network.
Why it matters
This work reveals that training vision-language models for robotics applications may inadvertently compromise their spatial reasoning abilities, particularly depth perception. Understanding these trade-offs is crucial for developing more robust AI systems that can both understand language about the visual world and execute physical actions effectively.
Understand the Science
⚠️ Preprint – Noch nicht peer-reviewed
Dieser Artikel wurde noch nicht von unabhängigen Experten begutachtet. Die Ergebnisse sind vorläufig und sollten mit Vorsicht interpretiert werden.
-cross
Abstract: How much of a vision-language model’s (VLM) spatial understanding remains after the action post-training process of building a vision-language-action model (VLA)? We probe depth perception, a primitive of spatiogeometric understanding, from every decoder layer of a weight-matched open-source base VLM/VLA pair: Molmo2-ER and MolmoAct2-LIBERO. First, the VLA decodes depth worse at every layer, a persistent gap we call the floor. Second, the degradation is not uniform: while the base VLM’s depth decodability improves through its final layers, the VLA’s collapses, an additional late-layer drop we call the cliff. We causally localize the cliff to late-layer MLP interference: ablating the late-layer MLP writes recovers the majority of the terminal decodability cliff, while matched attention ablations and the same intervention in the weight-matched base VLM produce no comparable recovery. A module-level decomposition explains this dissociation: the base VLM carries depth most accessibly in accumulated MLP writes, whereas action post-training collapses depth decodability in the late accumulated writes.
Source: From Recovery to Drop-off: How Action Post-training Reduces a VLM's Late-Layer Depth Decodability