AI Insight
This study investigates how diffusion models perform when operating in high-dimensional feature spaces created by visual encoders that have been fine-tuned for accurate image reconstruction. The researchers found that while fine-tuning encoders improves reconstruction quality, it paradoxically reduces the effective dimensionality of the representation space, creating geometric challenges for generation. They demonstrate that using direct data prediction (x0-prediction) instead of standard velocity prediction in flow matching models more efficiently focuses learning on the underlying signal structure, consistently improving text-to-image generation across multiple encoder architectures.
Why it matters
This work provides practical guidance for optimizing diffusion-based image generation systems that use pretrained visual encoders. By identifying the geometric limitations of high-dimensional latent spaces and proposing x0-prediction as a solution, the findings could lead to more efficient training and better quality outputs in text-to-image AI systems.
Understand the Science
⚠️ Preprint – Noch nicht peer-reviewed
Dieser Artikel wurde noch nicht von unabhängigen Experten begutachtet. Die Ergebnisse sind vorläufig und sollten mit Vorsicht interpretiert werden.
Abstract: Representation Autoencoders (RAEs) enable diffusion models to operate in the feature spaces of pretrained visual encoders. However, many off-the-shelf encoders are not optimized for faithful reconstruction, discarding fine-grained visual details. As expected, finetuning these encoders for image reconstruction recovers such details. However, perhaps counterintuitively, this procedure reduces the effective dimensionality of the resulting representation, and the altered geometry has downstream effects on generation. Specifically, we show that using the standard velocity prediction in flow matching in this high-dimensional space requires the model to fit orthogonal noise directions outside the low-dimensional signal manifold, making optimization inefficient. This motivates using the clean data parameterization ($boldsymbol{x}_{0}$-prediction) instead, which focuses learning on the underlying signal manifold. Across experiments with multiple strong-reconstruction encoders, we show that $boldsymbol{x}_{0}$-prediction consistently improves text-to-image generation performance.